2024-03-24T05:16:09.558 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T05:16:09.558 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --name teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi --verbose --archive /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617855 --description orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} -- /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617855/orig.config.yaml 2024-03-24T05:16:09.585 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T05:16:09.661 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617855 branch: squid description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} email: ceph-qa@ceph.com first_in_suite: false job_id: '7617855' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.1 - client.1 - prometheus.a - grafana.a - node-exporter.b - - mon.c - mgr.z - osd.2 - client.2 - node-exporter.c seed: 8903 sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 sleep_before_teardown: 0 subset: 44/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 targets: smithi023.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMlgx8TU2EZZJlDCNWuEJbkInhWgJsX/RQA0JH2FoA7VmjYF5vqLjUE3+rKq8eQXwWyvYa5UrI/dPohW+ASaLbo= smithi063.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPXnArZi4/aoVTQ0ccXCY8lD2PjU4Vhg/cgJLyN9uzbSqNRcpbsiWRuuJxr1JMN05J/Jwnp9V3VXmIHIwynPscU= smithi143.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKuztj9tdgI3H2Syq6QpP/oB/a+drUzJLKyUHlHCqTbap1yDLzVWSathX1KkSA85hlMXQ3upw1xLMrjI0a4CucE= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-24T05:16:09.662 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa; will attempt to use it 2024-03-24T05:16:09.662 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks 2024-03-24T05:16:09.662 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T05:16:09.664 INFO:teuthology.task.internal:Checking packages... 2024-03-24T05:16:09.685 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'db0330b1e4e2470d52b750e251e55a522b4f7d69' 2024-03-24T05:16:09.685 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T05:16:09.685 INFO:teuthology.packaging:ref: None 2024-03-24T05:16:09.685 INFO:teuthology.packaging:tag: None 2024-03-24T05:16:09.685 INFO:teuthology.packaging:branch: squid 2024-03-24T05:16:09.685 INFO:teuthology.packaging:sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T05:16:09.685 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-24T05:16:09.912 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1764.gf73f1dfd 2024-03-24T05:16:09.913 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T05:16:09.919 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T05:16:09.919 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T05:16:09.932 INFO:teuthology.task.internal:Saving configuration 2024-03-24T05:16:09.942 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T05:16:09.949 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T05:16:09.970 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617855', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 05:10:06.041341', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMlgx8TU2EZZJlDCNWuEJbkInhWgJsX/RQA0JH2FoA7VmjYF5vqLjUE3+rKq8eQXwWyvYa5UrI/dPohW+ASaLbo='} 2024-03-24T05:16:09.984 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617855', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 05:10:06.039004', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPXnArZi4/aoVTQ0ccXCY8lD2PjU4Vhg/cgJLyN9uzbSqNRcpbsiWRuuJxr1JMN05J/Jwnp9V3VXmIHIwynPscU='} 2024-03-24T05:16:10.003 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617855', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 05:10:06.043398', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKuztj9tdgI3H2Syq6QpP/oB/a+drUzJLKyUHlHCqTbap1yDLzVWSathX1KkSA85hlMXQ3upw1xLMrjI0a4CucE='} 2024-03-24T05:16:10.003 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T05:16:10.010 INFO:teuthology.task.internal:roles: ubuntu@smithi023.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-24T05:16:10.010 INFO:teuthology.task.internal:roles: ubuntu@smithi063.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.1', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-24T05:16:10.010 INFO:teuthology.task.internal:roles: ubuntu@smithi143.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'osd.2', 'client.2', 'node-exporter.c'] 2024-03-24T05:16:10.010 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T05:16:10.108 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff6826adf70>, signals=[15]) 2024-03-24T05:16:10.108 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T05:16:10.115 INFO:teuthology.task.internal:Opening connections... 2024-03-24T05:16:10.115 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi023.front.sepia.ceph.com 2024-03-24T05:16:10.117 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T05:16:10.223 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi063.front.sepia.ceph.com 2024-03-24T05:16:10.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T05:16:10.324 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi143.front.sepia.ceph.com 2024-03-24T05:16:10.325 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T05:16:10.412 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T05:16:10.421 DEBUG:teuthology.orchestra.run.smithi023:> uname -m 2024-03-24T05:16:10.439 INFO:teuthology.orchestra.run.smithi023.stdout:x86_64 2024-03-24T05:16:10.439 DEBUG:teuthology.orchestra.run.smithi023:> cat /etc/os-release 2024-03-24T05:16:10.497 INFO:teuthology.orchestra.run.smithi023.stdout:NAME="CentOS Stream" 2024-03-24T05:16:10.497 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION="9" 2024-03-24T05:16:10.498 INFO:teuthology.orchestra.run.smithi023.stdout:ID="centos" 2024-03-24T05:16:10.498 INFO:teuthology.orchestra.run.smithi023.stdout:ID_LIKE="rhel fedora" 2024-03-24T05:16:10.498 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_ID="9" 2024-03-24T05:16:10.498 INFO:teuthology.orchestra.run.smithi023.stdout:PLATFORM_ID="platform:el9" 2024-03-24T05:16:10.498 INFO:teuthology.orchestra.run.smithi023.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T05:16:10.498 INFO:teuthology.orchestra.run.smithi023.stdout:ANSI_COLOR="0;31" 2024-03-24T05:16:10.498 INFO:teuthology.orchestra.run.smithi023.stdout:LOGO="fedora-logo-icon" 2024-03-24T05:16:10.498 INFO:teuthology.orchestra.run.smithi023.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T05:16:10.498 INFO:teuthology.orchestra.run.smithi023.stdout:HOME_URL="https://centos.org/" 2024-03-24T05:16:10.498 INFO:teuthology.orchestra.run.smithi023.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T05:16:10.498 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T05:16:10.498 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T05:16:10.499 INFO:teuthology.lock.ops:Updating smithi023.front.sepia.ceph.com on lock server 2024-03-24T05:16:10.525 DEBUG:teuthology.orchestra.run.smithi063:> uname -m 2024-03-24T05:16:10.543 INFO:teuthology.orchestra.run.smithi063.stdout:x86_64 2024-03-24T05:16:10.543 DEBUG:teuthology.orchestra.run.smithi063:> cat /etc/os-release 2024-03-24T05:16:10.601 INFO:teuthology.orchestra.run.smithi063.stdout:NAME="CentOS Stream" 2024-03-24T05:16:10.601 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION="9" 2024-03-24T05:16:10.602 INFO:teuthology.orchestra.run.smithi063.stdout:ID="centos" 2024-03-24T05:16:10.602 INFO:teuthology.orchestra.run.smithi063.stdout:ID_LIKE="rhel fedora" 2024-03-24T05:16:10.602 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_ID="9" 2024-03-24T05:16:10.602 INFO:teuthology.orchestra.run.smithi063.stdout:PLATFORM_ID="platform:el9" 2024-03-24T05:16:10.602 INFO:teuthology.orchestra.run.smithi063.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T05:16:10.602 INFO:teuthology.orchestra.run.smithi063.stdout:ANSI_COLOR="0;31" 2024-03-24T05:16:10.602 INFO:teuthology.orchestra.run.smithi063.stdout:LOGO="fedora-logo-icon" 2024-03-24T05:16:10.602 INFO:teuthology.orchestra.run.smithi063.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T05:16:10.602 INFO:teuthology.orchestra.run.smithi063.stdout:HOME_URL="https://centos.org/" 2024-03-24T05:16:10.602 INFO:teuthology.orchestra.run.smithi063.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T05:16:10.602 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T05:16:10.602 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T05:16:10.603 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server 2024-03-24T05:16:10.624 DEBUG:teuthology.orchestra.run.smithi143:> uname -m 2024-03-24T05:16:10.642 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64 2024-03-24T05:16:10.643 DEBUG:teuthology.orchestra.run.smithi143:> cat /etc/os-release 2024-03-24T05:16:10.699 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="CentOS Stream" 2024-03-24T05:16:10.699 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="9" 2024-03-24T05:16:10.699 INFO:teuthology.orchestra.run.smithi143.stdout:ID="centos" 2024-03-24T05:16:10.700 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE="rhel fedora" 2024-03-24T05:16:10.700 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="9" 2024-03-24T05:16:10.700 INFO:teuthology.orchestra.run.smithi143.stdout:PLATFORM_ID="platform:el9" 2024-03-24T05:16:10.700 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T05:16:10.700 INFO:teuthology.orchestra.run.smithi143.stdout:ANSI_COLOR="0;31" 2024-03-24T05:16:10.700 INFO:teuthology.orchestra.run.smithi143.stdout:LOGO="fedora-logo-icon" 2024-03-24T05:16:10.700 INFO:teuthology.orchestra.run.smithi143.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T05:16:10.700 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://centos.org/" 2024-03-24T05:16:10.700 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T05:16:10.700 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T05:16:10.700 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T05:16:10.700 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server 2024-03-24T05:16:10.724 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T05:16:10.736 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T05:16:10.747 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T05:16:10.747 DEBUG:teuthology.orchestra.run.smithi023:> test '!' -e /home/ubuntu/cephtest 2024-03-24T05:16:10.750 DEBUG:teuthology.orchestra.run.smithi063:> test '!' -e /home/ubuntu/cephtest 2024-03-24T05:16:10.753 DEBUG:teuthology.orchestra.run.smithi143:> test '!' -e /home/ubuntu/cephtest 2024-03-24T05:16:10.768 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T05:16:10.777 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T05:16:10.778 DEBUG:teuthology.orchestra.run.smithi023:> test -z $(ls -A /var/lib/ceph) 2024-03-24T05:16:10.808 DEBUG:teuthology.orchestra.run.smithi063:> test -z $(ls -A /var/lib/ceph) 2024-03-24T05:16:10.810 DEBUG:teuthology.orchestra.run.smithi143:> test -z $(ls -A /var/lib/ceph) 2024-03-24T05:16:11.138 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T05:16:11.210 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T05:16:11.227 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-24T05:16:11.227 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-24T05:16:11.227 DEBUG:teuthology.orchestra.run.smithi023:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T05:16:11.228 DEBUG:teuthology.orchestra.run.smithi063:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T05:16:11.228 DEBUG:teuthology.orchestra.run.smithi143:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T05:16:11.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:16:11.246 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2024-03-24T05:16:11.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:16:11.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:16:11.248 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2024-03-24T05:16:11.248 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-03-24T05:16:11.303 INFO:teuthology.orchestra.run.smithi143.stdout:5.14.0-361.el9.x86_64 2024-03-24T05:16:11.303 INFO:teuthology.task.kernel:Running kernel on smithi143: 5.14.0-361.el9.x86_64 2024-03-24T05:16:11.303 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum install -y kernel 2024-03-24T05:16:11.307 INFO:teuthology.orchestra.run.smithi063.stdout:5.14.0-361.el9.x86_64 2024-03-24T05:16:11.307 INFO:teuthology.orchestra.run.smithi023.stdout:5.14.0-361.el9.x86_64 2024-03-24T05:16:11.307 INFO:teuthology.task.kernel:Running kernel on smithi063: 5.14.0-361.el9.x86_64 2024-03-24T05:16:11.307 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum install -y kernel 2024-03-24T05:16:11.308 INFO:teuthology.task.kernel:Running kernel on smithi023: 5.14.0-361.el9.x86_64 2024-03-24T05:16:11.308 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum install -y kernel 2024-03-24T05:16:13.423 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-24T05:16:13.449 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-24T05:16:13.591 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-24T05:16:14.933 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-24T05:16:15.023 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-9 - BaseOS 4.4 MB/s | 2.0 MB 00:00 2024-03-24T05:16:15.049 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2024-03-24T05:16:16.156 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-24T05:16:16.156 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-24T05:16:17.831 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 8.0 MB 00:02 2024-03-24T05:16:21.233 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - AppStream 4.7 MB/s | 19 MB 00:04 2024-03-24T05:16:25.352 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - CRB 9.6 MB/s | 6.1 MB 00:00 2024-03-24T05:16:27.359 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 16 kB 00:00 2024-03-24T05:16:28.377 INFO:teuthology.orchestra.run.smithi143.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-24T05:16:33.335 INFO:teuthology.orchestra.run.smithi143.stdout:lab-extras 44 kB/s | 1.7 kB 00:00 2024-03-24T05:16:34.789 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-24T05:16:35.409 INFO:teuthology.orchestra.run.smithi143.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T05:16:35.409 INFO:teuthology.orchestra.run.smithi143.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T05:16:35.509 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-24T05:16:35.519 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-24T05:16:35.519 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-03-24T05:16:35.520 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-24T05:16:35.520 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-24T05:16:35.520 INFO:teuthology.orchestra.run.smithi143.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T05:16:35.520 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-03-24T05:16:35.520 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T05:16:35.520 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T05:16:35.520 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T05:16:35.520 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T05:16:35.520 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-24T05:16:35.520 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-24T05:16:35.520 INFO:teuthology.orchestra.run.smithi143.stdout:Install 4 Packages 2024-03-24T05:16:35.520 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T05:16:35.521 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 76 M 2024-03-24T05:16:35.521 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 123 M 2024-03-24T05:16:35.521 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-24T05:16:35.728 INFO:teuthology.orchestra.run.smithi143.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 486 kB/s | 44 kB 00:00 2024-03-24T05:16:36.112 INFO:teuthology.orchestra.run.smithi143.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 31 MB/s | 15 MB 00:00 2024-03-24T05:16:36.479 INFO:teuthology.orchestra.run.smithi143.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 37 MB/s | 27 MB 00:00 2024-03-24T05:16:36.796 INFO:teuthology.orchestra.run.smithi143.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 29 MB/s | 33 MB 00:01 2024-03-24T05:16:36.796 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-24T05:16:36.796 INFO:teuthology.orchestra.run.smithi143.stdout:Total 59 MB/s | 76 MB 00:01 2024-03-24T05:16:37.457 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-24T05:16:37.810 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-24T05:16:37.810 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-24T05:16:37.879 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - AppStream 934 kB/s | 19 MB 00:20 2024-03-24T05:16:38.834 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-24T05:16:38.834 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-24T05:16:39.123 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.1 MB 00:00 2024-03-24T05:16:40.707 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-03-24T05:16:41.306 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-24T05:16:41.408 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 35 MB/s | 21 MB 00:00 2024-03-24T05:16:41.920 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T05:16:42.006 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T05:16:43.446 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - CRB 3.1 MB/s | 6.1 MB 00:01 2024-03-24T05:16:44.767 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T05:16:44.905 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T05:16:45.072 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-24T05:16:46.458 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-03-24T05:16:48.599 INFO:teuthology.orchestra.run.smithi063.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T05:16:48.599 INFO:teuthology.orchestra.run.smithi063.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T05:16:48.820 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-24T05:16:48.829 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-24T05:16:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-03-24T05:16:48.830 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-24T05:16:48.830 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-24T05:16:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T05:16:48.830 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-24T05:16:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T05:16:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T05:16:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T05:16:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-24T05:16:48.830 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-24T05:16:48.830 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-24T05:16:48.830 INFO:teuthology.orchestra.run.smithi063.stdout:Install 4 Packages 2024-03-24T05:16:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-24T05:16:48.831 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 76 M 2024-03-24T05:16:48.831 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 123 M 2024-03-24T05:16:48.831 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-24T05:16:49.618 INFO:teuthology.orchestra.run.smithi063.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 66 kB/s | 44 kB 00:00 2024-03-24T05:16:49.635 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T05:16:49.680 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T05:16:50.026 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 4.3 MB/s | 21 MB 00:04 2024-03-24T05:16:54.386 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T05:16:55.106 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-24T05:16:57.338 INFO:teuthology.orchestra.run.smithi023.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T05:16:57.338 INFO:teuthology.orchestra.run.smithi023.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T05:16:57.431 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2024-03-24T05:16:57.437 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2024-03-24T05:16:57.437 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2024-03-24T05:16:57.437 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2024-03-24T05:16:57.437 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2024-03-24T05:16:57.437 INFO:teuthology.orchestra.run.smithi023.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T05:16:57.437 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2024-03-24T05:16:57.438 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T05:16:57.438 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T05:16:57.438 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T05:16:57.438 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:16:57.438 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2024-03-24T05:16:57.438 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2024-03-24T05:16:57.438 INFO:teuthology.orchestra.run.smithi023.stdout:Install 4 Packages 2024-03-24T05:16:57.438 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:16:57.439 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 76 M 2024-03-24T05:16:57.439 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 123 M 2024-03-24T05:16:57.439 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2024-03-24T05:16:57.757 INFO:teuthology.orchestra.run.smithi023.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 240 kB/s | 44 kB 00:00 2024-03-24T05:17:00.091 INFO:teuthology.orchestra.run.smithi023.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 5.9 MB/s | 15 MB 00:02 2024-03-24T05:17:07.825 INFO:teuthology.orchestra.run.smithi063.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 808 kB/s | 15 MB 00:18 2024-03-24T05:17:17.230 INFO:teuthology.orchestra.run.smithi023.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 1.4 MB/s | 27 MB 00:19 2024-03-24T05:17:23.476 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T05:17:23.476 INFO:teuthology.orchestra.run.smithi143.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T05:17:23.476 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T05:17:23.520 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T05:17:23.878 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T05:17:23.878 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T05:17:23.878 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T05:17:23.878 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T05:17:24.222 INFO:teuthology.orchestra.run.smithi023.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 1.3 MB/s | 33 MB 00:26 2024-03-24T05:17:24.222 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2024-03-24T05:17:24.222 INFO:teuthology.orchestra.run.smithi023.stdout:Total 2.8 MB/s | 76 MB 00:26 2024-03-24T05:17:24.310 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T05:17:24.310 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T05:17:24.310 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-24T05:17:24.310 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T05:17:24.310 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T05:17:24.310 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T05:17:24.310 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T05:17:24.310 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T05:17:24.310 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-24T05:17:24.465 INFO:teuthology.orchestra.run.smithi063.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 807 kB/s | 27 MB 00:34 2024-03-24T05:17:24.627 DEBUG:teuthology.orchestra.run.smithi143:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T05:17:24.723 INFO:teuthology.orchestra.run.smithi143.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T05:17:24.723 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T05:17:24.723 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T05:17:24.724 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T05:17:24.724 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T05:17:24.724 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-24T05:17:24.724 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi143.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T05:17:24.724 DEBUG:teuthology.orchestra.run.smithi143:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T05:17:24.802 INFO:teuthology.orchestra.run.smithi143.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T05:17:24.803 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T05:17:24.803 DEBUG:teuthology.orchestra.run.smithi143:> sudo rpm -qi grub2-tools 2024-03-24T05:17:24.846 INFO:teuthology.orchestra.run.smithi143.stdout:Name : grub2-tools 2024-03-24T05:17:24.846 INFO:teuthology.orchestra.run.smithi143.stdout:Epoch : 1 2024-03-24T05:17:24.846 INFO:teuthology.orchestra.run.smithi143.stdout:Version : 2.06 2024-03-24T05:17:24.846 INFO:teuthology.orchestra.run.smithi143.stdout:Release : 68.el9 2024-03-24T05:17:24.846 INFO:teuthology.orchestra.run.smithi143.stdout:Architecture: x86_64 2024-03-24T05:17:24.846 INFO:teuthology.orchestra.run.smithi143.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T05:17:24.846 INFO:teuthology.orchestra.run.smithi143.stdout:Group : Unspecified 2024-03-24T05:17:24.846 INFO:teuthology.orchestra.run.smithi143.stdout:Size : 8264254 2024-03-24T05:17:24.846 INFO:teuthology.orchestra.run.smithi143.stdout:License : GPLv3+ 2024-03-24T05:17:24.847 INFO:teuthology.orchestra.run.smithi143.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T05:17:24.847 INFO:teuthology.orchestra.run.smithi143.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T05:17:24.847 INFO:teuthology.orchestra.run.smithi143.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T05:17:24.847 INFO:teuthology.orchestra.run.smithi143.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T05:17:24.847 INFO:teuthology.orchestra.run.smithi143.stdout:Packager : builder@centos.org 2024-03-24T05:17:24.847 INFO:teuthology.orchestra.run.smithi143.stdout:Vendor : CentOS 2024-03-24T05:17:24.847 INFO:teuthology.orchestra.run.smithi143.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T05:17:24.847 INFO:teuthology.orchestra.run.smithi143.stdout:Summary : Support tools for GRUB. 2024-03-24T05:17:24.847 INFO:teuthology.orchestra.run.smithi143.stdout:Description : 2024-03-24T05:17:24.847 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T05:17:24.847 INFO:teuthology.orchestra.run.smithi143.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T05:17:24.847 INFO:teuthology.orchestra.run.smithi143.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T05:17:24.847 INFO:teuthology.orchestra.run.smithi143.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T05:17:24.847 INFO:teuthology.orchestra.run.smithi143.stdout:hardware devices. 2024-03-24T05:17:24.847 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T05:17:24.848 INFO:teuthology.orchestra.run.smithi143.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T05:17:24.848 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T05:17:24.848 INFO:teuthology.task.kernel:Updating grub on smithi143 to boot 5.14.0-430.el9.x86_64 2024-03-24T05:17:24.848 DEBUG:teuthology.orchestra.run.smithi143:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T05:17:24.914 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2024-03-24T05:17:25.342 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2024-03-24T05:17:25.343 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2024-03-24T05:17:25.450 INFO:teuthology.orchestra.run.smithi143.stderr:Generating grub configuration file ... 2024-03-24T05:17:26.383 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2024-03-24T05:17:26.383 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2024-03-24T05:17:26.810 INFO:teuthology.orchestra.run.smithi143.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T05:17:26.843 INFO:teuthology.orchestra.run.smithi143.stderr:done 2024-03-24T05:17:26.845 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2024-03-24T05:17:26.860 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.0UikCqB4yY 2024-03-24T05:17:26.860 DEBUG:teuthology.orchestra.run.smithi143:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.0UikCqB4yY 2024-03-24T05:17:26.934 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.0UikCqB4yY 2024-03-24T05:17:27.075 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.0UikCqB4yY is 6KB 2024-03-24T05:17:27.094 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.0UikCqB4yY 2024-03-24T05:17:27.108 DEBUG:teuthology.orchestra.run.smithi143:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T05:17:27.180 INFO:teuthology.orchestra.run.smithi143.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T05:17:27.180 INFO:teuthology.orchestra.run.smithi143.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T05:17:27.180 INFO:teuthology.orchestra.run.smithi143.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T05:17:27.180 INFO:teuthology.orchestra.run.smithi143.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T05:17:27.182 DEBUG:teuthology.orchestra.run.smithi143:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T05:17:27.339 DEBUG:teuthology.orchestra.run.smithi143:> sudo shutdown -r now 2024-03-24T05:17:28.980 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2024-03-24T05:17:29.611 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T05:17:29.697 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T05:17:32.581 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T05:17:32.717 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T05:17:33.015 INFO:teuthology.orchestra.run.smithi063.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 777 kB/s | 33 MB 00:44 2024-03-24T05:17:33.015 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-24T05:17:33.015 INFO:teuthology.orchestra.run.smithi063.stdout:Total 1.7 MB/s | 76 MB 00:44 2024-03-24T05:17:33.702 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-24T05:17:34.120 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-24T05:17:34.120 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-24T05:17:35.161 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-24T05:17:35.162 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-24T05:17:37.772 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-24T05:17:37.814 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T05:17:37.866 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T05:17:38.418 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T05:17:38.505 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T05:17:41.390 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T05:17:41.524 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T05:17:42.992 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T05:17:46.603 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T05:17:46.666 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T05:17:51.772 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T05:17:57.342 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-24T05:17:57.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-24T05:17:57.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T05:18:13.864 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T05:18:13.864 INFO:teuthology.orchestra.run.smithi023.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T05:18:13.864 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:18:13.914 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T05:18:14.251 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T05:18:14.252 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T05:18:14.252 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T05:18:14.252 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T05:18:14.636 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T05:18:14.636 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:18:14.636 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2024-03-24T05:18:14.636 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T05:18:14.637 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T05:18:14.637 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T05:18:14.637 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T05:18:14.637 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:18:14.637 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2024-03-24T05:18:14.975 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T05:18:15.091 INFO:teuthology.orchestra.run.smithi023.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T05:18:15.091 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T05:18:15.091 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T05:18:15.091 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T05:18:15.091 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T05:18:15.092 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-24T05:18:15.092 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi023.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T05:18:15.092 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T05:18:15.175 INFO:teuthology.orchestra.run.smithi023.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T05:18:15.176 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T05:18:15.176 DEBUG:teuthology.orchestra.run.smithi023:> sudo rpm -qi grub2-tools 2024-03-24T05:18:15.213 INFO:teuthology.orchestra.run.smithi023.stdout:Name : grub2-tools 2024-03-24T05:18:15.213 INFO:teuthology.orchestra.run.smithi023.stdout:Epoch : 1 2024-03-24T05:18:15.214 INFO:teuthology.orchestra.run.smithi023.stdout:Version : 2.06 2024-03-24T05:18:15.214 INFO:teuthology.orchestra.run.smithi023.stdout:Release : 68.el9 2024-03-24T05:18:15.214 INFO:teuthology.orchestra.run.smithi023.stdout:Architecture: x86_64 2024-03-24T05:18:15.214 INFO:teuthology.orchestra.run.smithi023.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T05:18:15.214 INFO:teuthology.orchestra.run.smithi023.stdout:Group : Unspecified 2024-03-24T05:18:15.214 INFO:teuthology.orchestra.run.smithi023.stdout:Size : 8264254 2024-03-24T05:18:15.214 INFO:teuthology.orchestra.run.smithi023.stdout:License : GPLv3+ 2024-03-24T05:18:15.214 INFO:teuthology.orchestra.run.smithi023.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T05:18:15.214 INFO:teuthology.orchestra.run.smithi023.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T05:18:15.214 INFO:teuthology.orchestra.run.smithi023.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T05:18:15.215 INFO:teuthology.orchestra.run.smithi023.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T05:18:15.215 INFO:teuthology.orchestra.run.smithi023.stdout:Packager : builder@centos.org 2024-03-24T05:18:15.215 INFO:teuthology.orchestra.run.smithi023.stdout:Vendor : CentOS 2024-03-24T05:18:15.215 INFO:teuthology.orchestra.run.smithi023.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T05:18:15.215 INFO:teuthology.orchestra.run.smithi023.stdout:Summary : Support tools for GRUB. 2024-03-24T05:18:15.215 INFO:teuthology.orchestra.run.smithi023.stdout:Description : 2024-03-24T05:18:15.215 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:18:15.215 INFO:teuthology.orchestra.run.smithi023.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T05:18:15.215 INFO:teuthology.orchestra.run.smithi023.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T05:18:15.215 INFO:teuthology.orchestra.run.smithi023.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T05:18:15.215 INFO:teuthology.orchestra.run.smithi023.stdout:hardware devices. 2024-03-24T05:18:15.216 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:18:15.216 INFO:teuthology.orchestra.run.smithi023.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T05:18:15.217 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T05:18:15.217 INFO:teuthology.task.kernel:Updating grub on smithi023 to boot 5.14.0-430.el9.x86_64 2024-03-24T05:18:15.217 DEBUG:teuthology.orchestra.run.smithi023:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T05:18:15.838 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.143 2024-03-24T05:18:15.895 INFO:teuthology.orchestra.run.smithi023.stderr:Generating grub configuration file ... 2024-03-24T05:18:17.226 INFO:teuthology.orchestra.run.smithi023.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T05:18:17.270 INFO:teuthology.orchestra.run.smithi023.stderr:done 2024-03-24T05:18:17.272 DEBUG:teuthology.orchestra.run.smithi023:> mktemp 2024-03-24T05:18:17.288 INFO:teuthology.orchestra.run.smithi023.stdout:/tmp/tmp.LpZdNYUrkW 2024-03-24T05:18:17.288 DEBUG:teuthology.orchestra.run.smithi023:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.LpZdNYUrkW 2024-03-24T05:18:17.359 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0666 /tmp/tmp.LpZdNYUrkW 2024-03-24T05:18:17.543 DEBUG:teuthology.orchestra.remote:smithi023:/tmp/tmp.LpZdNYUrkW is 6KB 2024-03-24T05:18:17.594 DEBUG:teuthology.orchestra.run.smithi023:> rm -fr /tmp/tmp.LpZdNYUrkW 2024-03-24T05:18:17.609 DEBUG:teuthology.orchestra.run.smithi023:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T05:18:17.677 INFO:teuthology.orchestra.run.smithi023.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T05:18:17.677 INFO:teuthology.orchestra.run.smithi023.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T05:18:17.678 INFO:teuthology.orchestra.run.smithi023.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T05:18:17.678 INFO:teuthology.orchestra.run.smithi023.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T05:18:17.679 DEBUG:teuthology.orchestra.run.smithi023:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T05:18:17.831 DEBUG:teuthology.orchestra.run.smithi023:> sudo shutdown -r now 2024-03-24T05:18:22.675 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T05:18:22.675 INFO:teuthology.orchestra.run.smithi063.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T05:18:22.675 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-24T05:18:22.721 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T05:18:23.234 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T05:18:23.234 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T05:18:23.234 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T05:18:23.234 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T05:18:23.675 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T05:18:23.675 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-24T05:18:23.676 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-24T05:18:23.676 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T05:18:23.676 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T05:18:23.676 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T05:18:23.676 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T05:18:23.676 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-24T05:18:23.676 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-24T05:18:24.007 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T05:18:24.116 INFO:teuthology.orchestra.run.smithi063.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T05:18:24.116 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T05:18:24.116 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T05:18:24.116 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T05:18:24.116 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T05:18:24.116 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-24T05:18:24.116 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi063.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T05:18:24.117 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T05:18:24.200 INFO:teuthology.orchestra.run.smithi063.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T05:18:24.200 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T05:18:24.200 DEBUG:teuthology.orchestra.run.smithi063:> sudo rpm -qi grub2-tools 2024-03-24T05:18:24.238 INFO:teuthology.orchestra.run.smithi063.stdout:Name : grub2-tools 2024-03-24T05:18:24.238 INFO:teuthology.orchestra.run.smithi063.stdout:Epoch : 1 2024-03-24T05:18:24.238 INFO:teuthology.orchestra.run.smithi063.stdout:Version : 2.06 2024-03-24T05:18:24.238 INFO:teuthology.orchestra.run.smithi063.stdout:Release : 68.el9 2024-03-24T05:18:24.239 INFO:teuthology.orchestra.run.smithi063.stdout:Architecture: x86_64 2024-03-24T05:18:24.239 INFO:teuthology.orchestra.run.smithi063.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T05:18:24.239 INFO:teuthology.orchestra.run.smithi063.stdout:Group : Unspecified 2024-03-24T05:18:24.239 INFO:teuthology.orchestra.run.smithi063.stdout:Size : 8264254 2024-03-24T05:18:24.239 INFO:teuthology.orchestra.run.smithi063.stdout:License : GPLv3+ 2024-03-24T05:18:24.239 INFO:teuthology.orchestra.run.smithi063.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T05:18:24.239 INFO:teuthology.orchestra.run.smithi063.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T05:18:24.239 INFO:teuthology.orchestra.run.smithi063.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T05:18:24.239 INFO:teuthology.orchestra.run.smithi063.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T05:18:24.239 INFO:teuthology.orchestra.run.smithi063.stdout:Packager : builder@centos.org 2024-03-24T05:18:24.239 INFO:teuthology.orchestra.run.smithi063.stdout:Vendor : CentOS 2024-03-24T05:18:24.240 INFO:teuthology.orchestra.run.smithi063.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T05:18:24.240 INFO:teuthology.orchestra.run.smithi063.stdout:Summary : Support tools for GRUB. 2024-03-24T05:18:24.240 INFO:teuthology.orchestra.run.smithi063.stdout:Description : 2024-03-24T05:18:24.240 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-24T05:18:24.240 INFO:teuthology.orchestra.run.smithi063.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T05:18:24.240 INFO:teuthology.orchestra.run.smithi063.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T05:18:24.240 INFO:teuthology.orchestra.run.smithi063.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T05:18:24.240 INFO:teuthology.orchestra.run.smithi063.stdout:hardware devices. 2024-03-24T05:18:24.240 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-24T05:18:24.240 INFO:teuthology.orchestra.run.smithi063.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T05:18:24.241 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T05:18:24.241 INFO:teuthology.task.kernel:Updating grub on smithi063 to boot 5.14.0-430.el9.x86_64 2024-03-24T05:18:24.242 DEBUG:teuthology.orchestra.run.smithi063:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T05:18:24.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-24T05:18:24.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T05:18:24.860 INFO:teuthology.orchestra.run.smithi063.stderr:Generating grub configuration file ... 2024-03-24T05:18:26.263 INFO:teuthology.orchestra.run.smithi063.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T05:18:26.305 INFO:teuthology.orchestra.run.smithi063.stderr:done 2024-03-24T05:18:26.307 DEBUG:teuthology.orchestra.run.smithi063:> mktemp 2024-03-24T05:18:26.327 INFO:teuthology.orchestra.run.smithi063.stdout:/tmp/tmp.Qd6wYlzfl9 2024-03-24T05:18:26.327 DEBUG:teuthology.orchestra.run.smithi063:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Qd6wYlzfl9 2024-03-24T05:18:26.404 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0666 /tmp/tmp.Qd6wYlzfl9 2024-03-24T05:18:26.547 DEBUG:teuthology.orchestra.remote:smithi063:/tmp/tmp.Qd6wYlzfl9 is 6KB 2024-03-24T05:18:26.597 DEBUG:teuthology.orchestra.run.smithi063:> rm -fr /tmp/tmp.Qd6wYlzfl9 2024-03-24T05:18:26.612 DEBUG:teuthology.orchestra.run.smithi063:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T05:18:26.681 INFO:teuthology.orchestra.run.smithi063.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T05:18:26.681 INFO:teuthology.orchestra.run.smithi063.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T05:18:26.681 INFO:teuthology.orchestra.run.smithi063.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T05:18:26.681 INFO:teuthology.orchestra.run.smithi063.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T05:18:26.682 DEBUG:teuthology.orchestra.run.smithi063:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T05:18:26.845 DEBUG:teuthology.orchestra.run.smithi063:> sudo shutdown -r now 2024-03-24T05:18:27.902 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.143 2024-03-24T05:18:39.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-24T05:18:39.907 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T05:18:47.836 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-24T05:18:47.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-24T05:18:47.836 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T05:18:56.858 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-24T05:18:56.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-24T05:18:56.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T05:19:06.270 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-24T05:19:11.755 DEBUG:teuthology.orchestra.run.smithi143:> true 2024-03-24T05:19:12.131 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-24T05:19:12.131 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-430.el9.x86_64"... 2024-03-24T05:19:12.131 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2024-03-24T05:19:12.145 INFO:teuthology.orchestra.run.smithi143.stdout:5.14.0-430.el9.x86_64 2024-03-24T05:19:12.145 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T05:19:12.146 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T05:19:12.146 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T05:19:13.147 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-24T05:19:13.147 DEBUG:teuthology.orchestra.run.smithi143:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T05:19:13.233 INFO:teuthology.orchestra.run.smithi143.stdout:ttyS1 2024-03-24T05:19:13.267 DEBUG:teuthology.parallel:result is None 2024-03-24T05:19:15.231 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-03-24T05:19:15.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-24T05:19:15.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T05:19:18.336 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-24T05:19:24.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-24T05:19:24.235 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T05:19:27.294 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-03-24T05:19:30.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-24T05:19:30.338 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T05:19:33.406 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-24T05:19:39.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-24T05:19:39.296 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T05:19:48.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-24T05:19:48.410 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T05:19:51.486 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-24T05:19:57.730 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-03-24T05:20:09.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-24T05:20:09.489 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T05:20:12.732 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-24T05:20:12.733 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T05:20:13.428 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-03-24T05:20:13.848 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-24T05:20:13.848 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-24T05:20:13.848 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-03-24T05:20:13.862 INFO:teuthology.orchestra.run.smithi063.stdout:5.14.0-430.el9.x86_64 2024-03-24T05:20:13.863 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T05:20:13.863 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T05:20:13.863 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T05:20:14.864 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-24T05:20:14.864 DEBUG:teuthology.orchestra.run.smithi063:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T05:20:14.939 INFO:teuthology.orchestra.run.smithi063.stdout:ttyS1 2024-03-24T05:20:14.986 DEBUG:teuthology.parallel:result is None 2024-03-24T05:20:27.934 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-24T05:20:48.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-24T05:20:48.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T05:20:49.212 DEBUG:teuthology.orchestra.run.smithi023:> true 2024-03-24T05:20:49.606 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-24T05:20:49.607 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-24T05:20:49.607 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2024-03-24T05:20:49.623 INFO:teuthology.orchestra.run.smithi023.stdout:5.14.0-430.el9.x86_64 2024-03-24T05:20:49.623 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T05:20:49.623 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T05:20:49.623 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T05:20:50.624 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-24T05:20:50.624 DEBUG:teuthology.orchestra.run.smithi023:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T05:20:50.698 INFO:teuthology.orchestra.run.smithi023.stdout:ttyS1 2024-03-24T05:20:50.739 DEBUG:teuthology.parallel:result is None 2024-03-24T05:20:50.739 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T05:20:50.747 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T05:20:50.748 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T05:20:50.750 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T05:20:50.753 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T05:20:50.771 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T05:20:50.776 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T05:20:50.784 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T05:20:50.784 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T05:20:50.807 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T05:20:50.810 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T05:20:50.844 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T05:20:50.850 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T05:20:50.850 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T05:20:50.882 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T05:20:50.884 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T05:20:50.916 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T05:20:50.921 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T05:20:50.923 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T05:20:50.931 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T05:20:50.935 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T05:20:50.937 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T05:20:50.939 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T05:20:50.945 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T05:20:50.946 DEBUG:teuthology.orchestra.run.smithi023:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T05:20:50.975 DEBUG:teuthology.orchestra.run.smithi063:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T05:20:50.981 DEBUG:teuthology.orchestra.run.smithi143:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T05:20:51.012 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T05:20:51.021 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T05:20:51.021 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T05:20:51.044 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T05:20:51.051 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T05:20:51.068 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T05:20:51.135 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T05:20:51.190 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T05:20:51.268 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T05:20:51.317 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T05:20:51.317 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T05:20:51.384 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T05:20:51.424 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T05:20:51.485 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T05:20:51.566 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T05:20:51.615 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-24T05:20:51.615 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T05:20:51.684 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T05:20:51.725 DEBUG:teuthology.orchestra.run.smithi143:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T05:20:51.788 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T05:20:51.828 DEBUG:teuthology.orchestra.run.smithi143:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T05:20:51.875 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T05:20:51.875 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T05:20:51.944 DEBUG:teuthology.orchestra.run.smithi023:> sudo service rsyslog restart 2024-03-24T05:20:51.947 DEBUG:teuthology.orchestra.run.smithi063:> sudo service rsyslog restart 2024-03-24T05:20:51.949 DEBUG:teuthology.orchestra.run.smithi143:> sudo service rsyslog restart 2024-03-24T05:20:51.986 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T05:20:52.006 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T05:20:52.033 INFO:teuthology.orchestra.run.smithi143.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T05:20:52.369 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T05:20:52.377 INFO:teuthology.task.internal:Starting timer... 2024-03-24T05:20:52.377 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T05:20:52.386 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T05:20:52.394 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-24T05:20:52.395 DEBUG:teuthology.orchestra.run.smithi023:> sudo service auditd rotate 2024-03-24T05:20:52.457 INFO:teuthology.orchestra.run.smithi023.stdout:Rotating logs: 2024-03-24T05:20:52.459 DEBUG:teuthology.orchestra.run.smithi063:> sudo service auditd rotate 2024-03-24T05:20:52.514 INFO:teuthology.orchestra.run.smithi063.stdout:Rotating logs: 2024-03-24T05:20:52.516 DEBUG:teuthology.orchestra.run.smithi143:> sudo service auditd rotate 2024-03-24T05:20:52.576 INFO:teuthology.orchestra.run.smithi143.stdout:Rotating logs: 2024-03-24T05:20:52.577 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T05:20:52.577 DEBUG:teuthology.orchestra.run.smithi023:> /usr/sbin/getenforce 2024-03-24T05:20:52.601 INFO:teuthology.orchestra.run.smithi023.stdout:Permissive 2024-03-24T05:20:52.602 DEBUG:teuthology.orchestra.run.smithi063:> /usr/sbin/getenforce 2024-03-24T05:20:52.620 INFO:teuthology.orchestra.run.smithi063.stdout:Permissive 2024-03-24T05:20:52.620 DEBUG:teuthology.orchestra.run.smithi143:> /usr/sbin/getenforce 2024-03-24T05:20:52.645 INFO:teuthology.orchestra.run.smithi143.stdout:Permissive 2024-03-24T05:20:52.645 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi023.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi063.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi143.front.sepia.ceph.com': 'permissive'} 2024-03-24T05:20:52.646 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T05:20:52.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:20:52.675 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T05:20:52.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:20:52.704 DEBUG:teuthology.orchestra.run.smithi143:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T05:20:52.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:20:52.732 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T05:20:52.732 DEBUG:teuthology.orchestra.run.smithi023:> sudo /usr/sbin/setenforce permissive 2024-03-24T05:20:52.762 DEBUG:teuthology.orchestra.run.smithi063:> sudo /usr/sbin/setenforce permissive 2024-03-24T05:20:52.798 DEBUG:teuthology.orchestra.run.smithi143:> sudo /usr/sbin/setenforce permissive 2024-03-24T05:20:52.830 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T05:20:52.839 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-24T05:20:52.844 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-24T05:20:53.006 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T05:20:53.020 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T05:20:53.022 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi023.front.sepia.ceph.com,smithi063.front.sepia.ceph.com,smithi143.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T05:25:02.153 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi023.front.sepia.ceph.com'), Remote(name='ubuntu@smithi063.front.sepia.ceph.com'), Remote(name='ubuntu@smithi143.front.sepia.ceph.com')] 2024-03-24T05:25:02.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-24T05:25:02.155 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T05:25:02.247 DEBUG:teuthology.orchestra.run.smithi023:> true 2024-03-24T05:25:02.329 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-24T05:25:02.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-24T05:25:02.330 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T05:25:02.417 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-03-24T05:25:02.492 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-24T05:25:02.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-24T05:25:02.493 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T05:25:02.579 DEBUG:teuthology.orchestra.run.smithi143:> true 2024-03-24T05:25:02.657 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-24T05:25:02.658 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T05:25:02.668 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T05:25:02.668 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T05:25:02.669 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T05:25:02.671 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T05:25:02.671 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T05:25:02.673 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T05:25:02.674 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T05:25:02.702 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T05:25:02.702 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T05:25:02.718 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T05:25:02.719 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T05:25:02.729 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T05:25:02.746 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T05:25:02.747 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: ntpd: command not found 2024-03-24T05:25:02.749 INFO:teuthology.orchestra.run.smithi023.stderr:sudo: ntpd: command not found 2024-03-24T05:25:02.761 INFO:teuthology.orchestra.run.smithi063.stdout:506 Cannot talk to daemon 2024-03-24T05:25:02.763 INFO:teuthology.orchestra.run.smithi023.stdout:506 Cannot talk to daemon 2024-03-24T05:25:02.776 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: ntpd: command not found 2024-03-24T05:25:02.778 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T05:25:02.779 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T05:25:02.790 INFO:teuthology.orchestra.run.smithi143.stdout:506 Cannot talk to daemon 2024-03-24T05:25:02.794 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T05:25:02.796 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T05:25:02.807 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T05:25:02.823 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T05:25:02.848 INFO:teuthology.orchestra.run.smithi063.stderr:bash: line 1: ntpq: command not found 2024-03-24T05:25:02.851 INFO:teuthology.orchestra.run.smithi023.stderr:bash: line 1: ntpq: command not found 2024-03-24T05:25:02.851 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T05:25:02.851 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-03-24T05:25:02.851 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T05:25:02.851 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T05:25:02.851 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T05:25:02.852 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T05:25:02.854 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T05:25:02.855 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2024-03-24T05:25:02.855 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T05:25:02.855 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T05:25:02.855 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T05:25:02.855 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T05:25:02.882 INFO:teuthology.orchestra.run.smithi143.stderr:bash: line 1: ntpq: command not found 2024-03-24T05:25:02.886 INFO:teuthology.orchestra.run.smithi143.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T05:25:02.886 INFO:teuthology.orchestra.run.smithi143.stdout:=============================================================================== 2024-03-24T05:25:02.886 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T05:25:02.886 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T05:25:02.886 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T05:25:02.886 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T05:25:02.887 INFO:teuthology.run_tasks:Running task pexec... 2024-03-24T05:25:02.911 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-24T05:25:02.911 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi023.front.sepia.ceph.com 2024-03-24T05:25:02.911 DEBUG:teuthology.orchestra.run.smithi023:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-24T05:25:02.911 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi063.front.sepia.ceph.com 2024-03-24T05:25:02.912 DEBUG:teuthology.orchestra.run.smithi063:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-24T05:25:02.912 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi143.front.sepia.ceph.com 2024-03-24T05:25:02.912 DEBUG:teuthology.orchestra.run.smithi143:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-24T05:25:03.500 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:46 ago on Sun 24 Mar 2024 05:24:17 AM UTC. 2024-03-24T05:25:03.500 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:00:47 ago on Sun 24 Mar 2024 05:24:16 AM UTC. 2024-03-24T05:25:03.510 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:00:50 ago on Sun 24 Mar 2024 05:24:13 AM UTC. 2024-03-24T05:25:03.651 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2024-03-24T05:25:03.651 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-24T05:25:03.652 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2024-03-24T05:25:03.652 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2024-03-24T05:25:03.652 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2024-03-24T05:25:03.652 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2024-03-24T05:25:03.652 INFO:teuthology.orchestra.run.smithi023.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-24T05:25:03.652 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:25:03.653 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2024-03-24T05:25:03.653 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2024-03-24T05:25:03.653 INFO:teuthology.orchestra.run.smithi023.stdout:Install 1 Package 2024-03-24T05:25:03.653 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:25:03.657 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-24T05:25:03.658 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-24T05:25:03.658 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-03-24T05:25:03.658 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-24T05:25:03.659 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-24T05:25:03.659 INFO:teuthology.orchestra.run.smithi143.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-24T05:25:03.659 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T05:25:03.659 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-24T05:25:03.659 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-24T05:25:03.659 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-24T05:25:03.659 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T05:25:03.660 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-24T05:25:03.660 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-03-24T05:25:03.660 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-24T05:25:03.660 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-24T05:25:03.660 INFO:teuthology.orchestra.run.smithi063.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-24T05:25:03.660 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-24T05:25:03.660 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-24T05:25:03.660 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-24T05:25:03.661 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-24T05:25:03.661 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-24T05:25:03.662 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 3.1 M 2024-03-24T05:25:03.662 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 9.8 M 2024-03-24T05:25:03.662 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2024-03-24T05:25:03.663 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 3.1 M 2024-03-24T05:25:03.663 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 9.8 M 2024-03-24T05:25:03.663 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-24T05:25:03.671 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 3.1 M 2024-03-24T05:25:03.671 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 9.8 M 2024-03-24T05:25:03.672 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-24T05:25:03.981 INFO:teuthology.orchestra.run.smithi143.stdout:runc-1.1.12-2.el9.x86_64.rpm 12 MB/s | 3.1 MB 00:00 2024-03-24T05:25:03.981 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-24T05:25:03.981 INFO:teuthology.orchestra.run.smithi143.stdout:Total 10 MB/s | 3.1 MB 00:00 2024-03-24T05:25:04.018 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-24T05:25:04.023 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-24T05:25:04.023 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-24T05:25:04.104 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-24T05:25:04.104 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-24T05:25:04.335 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-24T05:25:04.435 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T05:25:04.456 INFO:teuthology.orchestra.run.smithi023.stdout:runc-1.1.12-2.el9.x86_64.rpm 4.3 MB/s | 3.1 MB 00:00 2024-03-24T05:25:04.456 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2024-03-24T05:25:04.457 INFO:teuthology.orchestra.run.smithi023.stdout:Total 3.9 MB/s | 3.1 MB 00:00 2024-03-24T05:25:04.494 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2024-03-24T05:25:04.500 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2024-03-24T05:25:04.500 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2024-03-24T05:25:04.583 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2024-03-24T05:25:04.583 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2024-03-24T05:25:04.702 INFO:teuthology.orchestra.run.smithi063.stdout:runc-1.1.12-2.el9.x86_64.rpm 3.4 MB/s | 3.1 MB 00:00 2024-03-24T05:25:04.702 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-24T05:25:04.702 INFO:teuthology.orchestra.run.smithi063.stdout:Total 3.0 MB/s | 3.1 MB 00:01 2024-03-24T05:25:04.740 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-24T05:25:04.746 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-24T05:25:04.746 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-24T05:25:04.793 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2024-03-24T05:25:04.830 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-24T05:25:04.830 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-24T05:25:04.897 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T05:25:05.019 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T05:25:05.041 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-24T05:25:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T05:25:05.306 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T05:25:05.306 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T05:25:05.306 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-24T05:25:05.307 INFO:teuthology.orchestra.run.smithi143.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-24T05:25:05.307 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T05:25:05.307 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-24T05:25:05.400 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T05:25:05.562 DEBUG:teuthology.parallel:result is None 2024-03-24T05:25:05.637 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T05:25:05.685 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T05:25:05.685 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:25:05.685 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2024-03-24T05:25:05.685 INFO:teuthology.orchestra.run.smithi023.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-24T05:25:05.686 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:25:05.686 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2024-03-24T05:25:05.927 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T05:25:05.927 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-24T05:25:05.928 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-24T05:25:05.928 INFO:teuthology.orchestra.run.smithi063.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-24T05:25:05.928 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-24T05:25:05.928 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-24T05:25:05.951 DEBUG:teuthology.parallel:result is None 2024-03-24T05:25:06.181 DEBUG:teuthology.parallel:result is None 2024-03-24T05:25:06.181 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-24T05:25:06.192 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-24T05:25:06.192 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T05:25:06.192 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T05:25:06.207 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T05:25:06.208 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_1 2024-03-24T05:25:06.262 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T05:25:06.262 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T05:25:06.262 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-24T05:25:06.262 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T05:25:06.262 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T05:25:06.262 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-24 05:25:01.165496445 +0000 2024-03-24T05:25:06.262 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-24 05:24:39.724670768 +0000 2024-03-24T05:25:06.263 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-24 05:24:39.724670768 +0000 2024-03-24T05:25:06.263 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-24 05:24:39.724670768 +0000 2024-03-24T05:25:06.263 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T05:25:06.328 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-24T05:25:06.328 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-24T05:25:06.328 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000302309 s, 1.7 MB/s 2024-03-24T05:25:06.329 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T05:25:06.385 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_2 2024-03-24T05:25:06.439 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T05:25:06.439 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T05:25:06.439 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-24T05:25:06.439 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T05:25:06.439 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T05:25:06.440 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-24 05:25:01.165496445 +0000 2024-03-24T05:25:06.440 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-24 05:24:40.067667977 +0000 2024-03-24T05:25:06.440 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-24 05:24:40.067667977 +0000 2024-03-24T05:25:06.440 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-24 05:24:40.067667977 +0000 2024-03-24T05:25:06.440 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T05:25:06.505 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-24T05:25:06.505 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-24T05:25:06.505 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000250614 s, 2.0 MB/s 2024-03-24T05:25:06.506 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T05:25:06.561 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_3 2024-03-24T05:25:06.616 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T05:25:06.616 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T05:25:06.616 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-24T05:25:06.616 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T05:25:06.616 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T05:25:06.616 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-24 05:25:01.165496445 +0000 2024-03-24T05:25:06.616 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-24 05:24:40.411665177 +0000 2024-03-24T05:25:06.616 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-24 05:24:40.411665177 +0000 2024-03-24T05:25:06.616 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-24 05:24:40.411665177 +0000 2024-03-24T05:25:06.616 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T05:25:06.680 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-24T05:25:06.681 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-24T05:25:06.681 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.00026576 s, 1.9 MB/s 2024-03-24T05:25:06.682 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T05:25:06.737 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_4 2024-03-24T05:25:06.792 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T05:25:06.792 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T05:25:06.792 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-24T05:25:06.792 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T05:25:06.792 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T05:25:06.792 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-24 05:25:01.165496445 +0000 2024-03-24T05:25:06.792 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-24 05:24:40.750662418 +0000 2024-03-24T05:25:06.792 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-24 05:24:40.750662418 +0000 2024-03-24T05:25:06.792 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-24 05:24:40.750662418 +0000 2024-03-24T05:25:06.792 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T05:25:06.856 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-24T05:25:06.857 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-24T05:25:06.857 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000273167 s, 1.9 MB/s 2024-03-24T05:25:06.858 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T05:25:06.912 DEBUG:teuthology.orchestra.run.smithi023:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-24T05:25:07.058 INFO:teuthology.orchestra.run.smithi023.stdout:loop 2024-03-24T05:25:07.060 INFO:tasks.nvme_loop:Connecting nvme_loop smithi023:/dev/vg_nvme/lv_1... 2024-03-24T05:25:07.060 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-24T05:25:07.097 INFO:teuthology.orchestra.run.smithi023.stdout:1 2024-03-24T05:25:07.134 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/vg_nvme/lv_11 2024-03-24T05:25:07.164 INFO:tasks.nvme_loop:Connecting nvme_loop smithi023:/dev/vg_nvme/lv_2... 2024-03-24T05:25:07.164 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-24T05:25:07.241 INFO:teuthology.orchestra.run.smithi023.stdout:1 2024-03-24T05:25:07.281 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/vg_nvme/lv_21 2024-03-24T05:25:07.323 INFO:tasks.nvme_loop:Connecting nvme_loop smithi023:/dev/vg_nvme/lv_3... 2024-03-24T05:25:07.323 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-24T05:25:07.401 INFO:teuthology.orchestra.run.smithi023.stdout:1 2024-03-24T05:25:07.439 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/vg_nvme/lv_31 2024-03-24T05:25:07.474 INFO:tasks.nvme_loop:Connecting nvme_loop smithi023:/dev/vg_nvme/lv_4... 2024-03-24T05:25:07.474 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-24T05:25:07.552 INFO:teuthology.orchestra.run.smithi023.stdout:1 2024-03-24T05:25:07.589 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/vg_nvme/lv_41 2024-03-24T05:25:07.629 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T05:25:07.629 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T05:25:07.683 DEBUG:teuthology.orchestra.run.smithi023:> sudo nvme list 2024-03-24T05:25:07.748 INFO:teuthology.orchestra.run.smithi023.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-24T05:25:07.748 INFO:teuthology.orchestra.run.smithi023.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-24T05:25:07.748 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/nvme0n1 BTLE736103B14P0KGN INTEL SSDPEDKE040T7 1 4.00 TB / 4.00 TB 512 B + 0 B QDV1013D 2024-03-24T05:25:07.748 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/nvme1n1 8549904116af059a0488 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T05:25:07.748 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/nvme2n1 28689d8b43718627a57c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T05:25:07.748 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/nvme3n1 b7d9cdebba36738fb371 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T05:25:07.748 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/nvme4n1 cc13a3e672a650c05812 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T05:25:07.749 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-24T05:25:07.749 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T05:25:07.749 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/scratch_devs 2024-03-24T05:25:07.816 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-24T05:25:07.816 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T05:25:07.835 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T05:25:07.835 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_1 2024-03-24T05:25:07.892 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T05:25:07.892 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T05:25:07.892 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-24T05:25:07.892 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T05:25:07.892 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T05:25:07.892 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-24 05:25:01.239231762 +0000 2024-03-24T05:25:07.892 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-24 05:24:40.549465118 +0000 2024-03-24T05:25:07.892 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-24 05:24:40.549465118 +0000 2024-03-24T05:25:07.892 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-24 05:24:40.549465118 +0000 2024-03-24T05:25:07.892 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T05:25:07.961 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-24T05:25:07.961 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-24T05:25:07.961 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000252732 s, 2.0 MB/s 2024-03-24T05:25:07.963 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T05:25:08.021 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_2 2024-03-24T05:25:08.077 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T05:25:08.077 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T05:25:08.077 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-24T05:25:08.077 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T05:25:08.077 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T05:25:08.078 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-24 05:25:01.239231762 +0000 2024-03-24T05:25:08.078 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-24 05:24:40.892461250 +0000 2024-03-24T05:25:08.078 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-24 05:24:40.892461250 +0000 2024-03-24T05:25:08.078 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-24 05:24:40.892461250 +0000 2024-03-24T05:25:08.078 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T05:25:08.144 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-24T05:25:08.144 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-24T05:25:08.144 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000197996 s, 2.6 MB/s 2024-03-24T05:25:08.146 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T05:25:08.202 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_3 2024-03-24T05:25:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T05:25:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T05:25:08.257 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-24T05:25:08.257 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T05:25:08.258 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T05:25:08.258 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-24 05:25:01.239231762 +0000 2024-03-24T05:25:08.258 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-24 05:24:41.238457348 +0000 2024-03-24T05:25:08.258 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-24 05:24:41.238457348 +0000 2024-03-24T05:25:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-24 05:24:41.238457348 +0000 2024-03-24T05:25:08.258 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T05:25:08.323 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-24T05:25:08.324 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-24T05:25:08.324 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000245832 s, 2.1 MB/s 2024-03-24T05:25:08.325 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T05:25:08.381 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_4 2024-03-24T05:25:08.436 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T05:25:08.436 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T05:25:08.436 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-24T05:25:08.436 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T05:25:08.436 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T05:25:08.436 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-24 05:25:01.239231762 +0000 2024-03-24T05:25:08.436 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-24 05:24:41.579453501 +0000 2024-03-24T05:25:08.436 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-24 05:24:41.579453501 +0000 2024-03-24T05:25:08.436 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-24 05:24:41.579453501 +0000 2024-03-24T05:25:08.437 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T05:25:08.502 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-24T05:25:08.502 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-24T05:25:08.502 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000185458 s, 2.8 MB/s 2024-03-24T05:25:08.503 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T05:25:08.558 DEBUG:teuthology.orchestra.run.smithi063:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-24T05:25:08.705 INFO:teuthology.orchestra.run.smithi063.stdout:loop 2024-03-24T05:25:08.707 INFO:tasks.nvme_loop:Connecting nvme_loop smithi063:/dev/vg_nvme/lv_1... 2024-03-24T05:25:08.707 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-24T05:25:08.743 INFO:teuthology.orchestra.run.smithi063.stdout:1 2024-03-24T05:25:08.781 INFO:teuthology.orchestra.run.smithi063.stdout:/dev/vg_nvme/lv_11 2024-03-24T05:25:08.822 INFO:tasks.nvme_loop:Connecting nvme_loop smithi063:/dev/vg_nvme/lv_2... 2024-03-24T05:25:08.822 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-24T05:25:08.902 INFO:teuthology.orchestra.run.smithi063.stdout:1 2024-03-24T05:25:08.941 INFO:teuthology.orchestra.run.smithi063.stdout:/dev/vg_nvme/lv_21 2024-03-24T05:25:08.970 INFO:tasks.nvme_loop:Connecting nvme_loop smithi063:/dev/vg_nvme/lv_3... 2024-03-24T05:25:08.970 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-24T05:25:09.054 INFO:teuthology.orchestra.run.smithi063.stdout:1 2024-03-24T05:25:09.093 INFO:teuthology.orchestra.run.smithi063.stdout:/dev/vg_nvme/lv_31 2024-03-24T05:25:09.122 INFO:tasks.nvme_loop:Connecting nvme_loop smithi063:/dev/vg_nvme/lv_4... 2024-03-24T05:25:09.122 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-24T05:25:09.206 INFO:teuthology.orchestra.run.smithi063.stdout:1 2024-03-24T05:25:09.246 INFO:teuthology.orchestra.run.smithi063.stdout:/dev/vg_nvme/lv_41 2024-03-24T05:25:09.275 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-24T05:25:09.275 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T05:25:09.330 DEBUG:teuthology.orchestra.run.smithi063:> sudo nvme list 2024-03-24T05:25:09.396 INFO:teuthology.orchestra.run.smithi063.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-24T05:25:09.396 INFO:teuthology.orchestra.run.smithi063.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-24T05:25:09.396 INFO:teuthology.orchestra.run.smithi063.stdout:/dev/nvme0n1 CVFT5344005B400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-24T05:25:09.396 INFO:teuthology.orchestra.run.smithi063.stdout:/dev/nvme1n1 1162f4684f88f110d8c7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T05:25:09.396 INFO:teuthology.orchestra.run.smithi063.stdout:/dev/nvme2n1 ff402ff699d2c9ece9a6 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T05:25:09.396 INFO:teuthology.orchestra.run.smithi063.stdout:/dev/nvme3n1 188d14ce55301b58156c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T05:25:09.396 INFO:teuthology.orchestra.run.smithi063.stdout:/dev/nvme4n1 144efd7c913394e7e3ec Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T05:25:09.397 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-24T05:25:09.397 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-24T05:25:09.397 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/scratch_devs 2024-03-24T05:25:09.465 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T05:25:09.465 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T05:25:09.483 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T05:25:09.483 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_1 2024-03-24T05:25:09.540 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T05:25:09.540 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T05:25:09.540 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-24T05:25:09.540 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T05:25:09.540 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T05:25:09.541 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-24 05:25:01.193073950 +0000 2024-03-24T05:25:09.541 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-24 05:24:37.789700948 +0000 2024-03-24T05:25:09.541 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-24 05:24:37.789700948 +0000 2024-03-24T05:25:09.541 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-03-24 05:24:37.789700948 +0000 2024-03-24T05:25:09.541 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T05:25:09.609 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-24T05:25:09.609 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-24T05:25:09.609 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.00021757 s, 2.4 MB/s 2024-03-24T05:25:09.610 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T05:25:09.669 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_2 2024-03-24T05:25:09.725 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T05:25:09.725 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T05:25:09.725 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-24T05:25:09.725 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T05:25:09.725 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T05:25:09.725 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-24 05:25:01.193073950 +0000 2024-03-24T05:25:09.725 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-24 05:24:38.125691946 +0000 2024-03-24T05:25:09.726 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-24 05:24:38.125691946 +0000 2024-03-24T05:25:09.726 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-03-24 05:24:38.125691946 +0000 2024-03-24T05:25:09.726 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T05:25:09.793 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-24T05:25:09.793 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-24T05:25:09.793 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000198149 s, 2.6 MB/s 2024-03-24T05:25:09.795 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T05:25:09.852 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_3 2024-03-24T05:25:09.908 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T05:25:09.908 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T05:25:09.908 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-24T05:25:09.908 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T05:25:09.908 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T05:25:09.908 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-24 05:25:01.193073950 +0000 2024-03-24T05:25:09.908 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-24 05:24:38.457683052 +0000 2024-03-24T05:25:09.908 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-24 05:24:38.457683052 +0000 2024-03-24T05:25:09.908 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-03-24 05:24:38.457683052 +0000 2024-03-24T05:25:09.909 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T05:25:09.975 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-24T05:25:09.975 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-24T05:25:09.975 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000239334 s, 2.1 MB/s 2024-03-24T05:25:09.977 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T05:25:10.034 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_4 2024-03-24T05:25:10.089 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T05:25:10.090 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T05:25:10.090 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-24T05:25:10.090 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T05:25:10.090 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T05:25:10.090 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-24 05:25:01.193073950 +0000 2024-03-24T05:25:10.090 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-24 05:24:38.795673996 +0000 2024-03-24T05:25:10.090 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-24 05:24:38.795673996 +0000 2024-03-24T05:25:10.090 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-03-24 05:24:38.795673996 +0000 2024-03-24T05:25:10.090 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T05:25:10.156 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-24T05:25:10.156 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-24T05:25:10.156 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000213768 s, 2.4 MB/s 2024-03-24T05:25:10.157 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T05:25:10.214 DEBUG:teuthology.orchestra.run.smithi143:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-24T05:25:10.359 INFO:teuthology.orchestra.run.smithi143.stdout:loop 2024-03-24T05:25:10.361 INFO:tasks.nvme_loop:Connecting nvme_loop smithi143:/dev/vg_nvme/lv_1... 2024-03-24T05:25:10.361 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-24T05:25:10.399 INFO:teuthology.orchestra.run.smithi143.stdout:1 2024-03-24T05:25:10.436 INFO:teuthology.orchestra.run.smithi143.stdout:/dev/vg_nvme/lv_11 2024-03-24T05:25:10.465 INFO:tasks.nvme_loop:Connecting nvme_loop smithi143:/dev/vg_nvme/lv_2... 2024-03-24T05:25:10.465 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-24T05:25:10.548 INFO:teuthology.orchestra.run.smithi143.stdout:1 2024-03-24T05:25:10.588 INFO:teuthology.orchestra.run.smithi143.stdout:/dev/vg_nvme/lv_21 2024-03-24T05:25:10.618 INFO:tasks.nvme_loop:Connecting nvme_loop smithi143:/dev/vg_nvme/lv_3... 2024-03-24T05:25:10.618 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-24T05:25:10.697 INFO:teuthology.orchestra.run.smithi143.stdout:1 2024-03-24T05:25:10.737 INFO:teuthology.orchestra.run.smithi143.stdout:/dev/vg_nvme/lv_31 2024-03-24T05:25:10.766 INFO:tasks.nvme_loop:Connecting nvme_loop smithi143:/dev/vg_nvme/lv_4... 2024-03-24T05:25:10.766 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-24T05:25:10.844 INFO:teuthology.orchestra.run.smithi143.stdout:1 2024-03-24T05:25:10.884 INFO:teuthology.orchestra.run.smithi143.stdout:/dev/vg_nvme/lv_41 2024-03-24T05:25:10.915 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T05:25:10.915 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T05:25:10.976 DEBUG:teuthology.orchestra.run.smithi143:> sudo nvme list 2024-03-24T05:25:11.043 INFO:teuthology.orchestra.run.smithi143.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-24T05:25:11.043 INFO:teuthology.orchestra.run.smithi143.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-24T05:25:11.043 INFO:teuthology.orchestra.run.smithi143.stdout:/dev/nvme0n1 CVFT62330073400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-24T05:25:11.043 INFO:teuthology.orchestra.run.smithi143.stdout:/dev/nvme1n1 948f005413c860afb7e4 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T05:25:11.043 INFO:teuthology.orchestra.run.smithi143.stdout:/dev/nvme2n1 55da23698e9b0746f0c8 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T05:25:11.043 INFO:teuthology.orchestra.run.smithi143.stdout:/dev/nvme3n1 1f4e4e16c9dfacc88dba Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T05:25:11.043 INFO:teuthology.orchestra.run.smithi143.stdout:/dev/nvme4n1 8af335e728b9053b18b9 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T05:25:11.044 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-24T05:25:11.044 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T05:25:11.044 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/scratch_devs 2024-03-24T05:25:11.112 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-24T05:25:11.197 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'global': {'mon election default strategy': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'} 2024-03-24T05:25:11.197 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T05:25:11.198 INFO:tasks.cephadm:Cluster fsid is e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:25:11.198 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-24T05:25:11.198 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.23', 'mon.b': '172.21.15.63', 'mon.c': '172.21.15.143'} 2024-03-24T05:25:11.198 INFO:tasks.cephadm:First mon is mon.a on smithi023 2024-03-24T05:25:11.198 INFO:tasks.cephadm:First mgr is y 2024-03-24T05:25:11.198 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-24T05:25:11.199 DEBUG:teuthology.orchestra.run.smithi023:> sudo hostname $(hostname -s) 2024-03-24T05:25:11.230 DEBUG:teuthology.orchestra.run.smithi063:> sudo hostname $(hostname -s) 2024-03-24T05:25:11.262 DEBUG:teuthology.orchestra.run.smithi143:> sudo hostname $(hostname -s) 2024-03-24T05:25:11.288 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-24T05:25:11.289 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T05:25:11.435 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77704/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1667-gdb0330b1', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1667.gdb0330b1'}, 'url': 'https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-16 11:55:36.797747', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-24T05:25:11.576 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref squid, sha1 db0330b1e4e2470d52b750e251e55a522b4f7d69 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T05:25:11.577 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/x86_64/flavors/default/cephadm 2024-03-24T05:25:11.577 DEBUG:teuthology.orchestra.run.smithi023:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T05:25:12.027 INFO:teuthology.orchestra.run.smithi023.stdout:-rw-r--r--. 1 ubuntu ubuntu 766879 Mar 24 05:25 /home/ubuntu/cephtest/cephadm 2024-03-24T05:25:12.027 DEBUG:teuthology.orchestra.run.smithi063:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T05:25:12.476 INFO:teuthology.orchestra.run.smithi063.stdout:-rw-r--r--. 1 ubuntu ubuntu 766879 Mar 24 05:25 /home/ubuntu/cephtest/cephadm 2024-03-24T05:25:12.477 DEBUG:teuthology.orchestra.run.smithi143:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T05:25:12.924 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r--. 1 ubuntu ubuntu 766879 Mar 24 05:25 /home/ubuntu/cephtest/cephadm 2024-03-24T05:25:12.925 DEBUG:teuthology.orchestra.run.smithi023:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T05:25:12.943 DEBUG:teuthology.orchestra.run.smithi063:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T05:25:12.961 DEBUG:teuthology.orchestra.run.smithi143:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T05:25:12.991 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 on all hosts... 2024-03-24T05:25:12.991 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 pull 2024-03-24T05:25:12.993 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 pull 2024-03-24T05:25:13.003 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 pull 2024-03-24T05:25:13.763 INFO:teuthology.orchestra.run.smithi063.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T05:25:13.803 INFO:teuthology.orchestra.run.smithi023.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T05:25:13.907 INFO:teuthology.orchestra.run.smithi143.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T05:25:41.506 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-24T05:25:41.506 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_version": "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)", 2024-03-24T05:25:41.506 INFO:teuthology.orchestra.run.smithi143.stdout: "image_id": "4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d", 2024-03-24T05:25:41.506 INFO:teuthology.orchestra.run.smithi143.stdout: "repo_digests": [ 2024-03-24T05:25:41.506 INFO:teuthology.orchestra.run.smithi143.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10" 2024-03-24T05:25:41.506 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-24T05:25:41.507 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-24T05:25:57.699 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-24T05:25:57.699 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_version": "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)", 2024-03-24T05:25:57.699 INFO:teuthology.orchestra.run.smithi063.stdout: "image_id": "4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d", 2024-03-24T05:25:57.699 INFO:teuthology.orchestra.run.smithi063.stdout: "repo_digests": [ 2024-03-24T05:25:57.699 INFO:teuthology.orchestra.run.smithi063.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10" 2024-03-24T05:25:57.699 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-03-24T05:25:57.699 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-24T05:26:01.656 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T05:26:01.657 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_version": "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)", 2024-03-24T05:26:01.657 INFO:teuthology.orchestra.run.smithi023.stdout: "image_id": "4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d", 2024-03-24T05:26:01.657 INFO:teuthology.orchestra.run.smithi023.stdout: "repo_digests": [ 2024-03-24T05:26:01.657 INFO:teuthology.orchestra.run.smithi023.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10" 2024-03-24T05:26:01.657 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-24T05:26:01.657 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T05:26:01.681 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /etc/ceph 2024-03-24T05:26:01.727 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /etc/ceph 2024-03-24T05:26:01.761 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /etc/ceph 2024-03-24T05:26:01.796 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 777 /etc/ceph 2024-03-24T05:26:01.823 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 777 /etc/ceph 2024-03-24T05:26:01.854 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 777 /etc/ceph 2024-03-24T05:26:01.885 INFO:tasks.cephadm:Writing seed config... 2024-03-24T05:26:01.886 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-24T05:26:01.887 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-24T05:26:01.887 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-03-24T05:26:01.887 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-24T05:26:01.887 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-24T05:26:01.887 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-24T05:26:01.887 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-24T05:26:01.887 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-24T05:26:01.887 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-24T05:26:01.887 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-24T05:26:01.888 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T05:26:01.888 DEBUG:teuthology.orchestra.run.smithi023:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-24T05:26:01.902 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e2aa55c2-e99e-11ee-95cd-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-24T05:26:01.902 DEBUG:teuthology.orchestra.run.smithi023:mon.a> sudo journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mon.a.service 2024-03-24T05:26:01.945 DEBUG:teuthology.orchestra.run.smithi023:mgr.y> sudo journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mgr.y.service 2024-03-24T05:26:01.987 INFO:tasks.cephadm:Bootstrapping... 2024-03-24T05:26:01.988 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 -v bootstrap --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.23 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-24T05:26:02.181 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2024-03-24T05:26:02.181 INFO:teuthology.orchestra.run.smithi023.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69', '-v', 'bootstrap', '--fsid', 'e2aa55c2-e99e-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.23', '--skip-admin-label'] 2024-03-24T05:26:02.207 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T05:26:02.207 INFO:teuthology.orchestra.run.smithi023.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-24T05:26:02.208 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying podman|docker is present... 2024-03-24T05:26:02.230 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T05:26:02.230 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying lvm2 is present... 2024-03-24T05:26:02.230 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying time synchronization is in place... 2024-03-24T05:26:02.237 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T05:26:02.237 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T05:26:02.243 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T05:26:02.243 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-24T05:26:02.250 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout enabled 2024-03-24T05:26:02.255 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout active 2024-03-24T05:26:02.255 INFO:teuthology.orchestra.run.smithi023.stdout:Unit chronyd.service is enabled and running 2024-03-24T05:26:02.256 INFO:teuthology.orchestra.run.smithi023.stdout:Repeating the final host check... 2024-03-24T05:26:02.278 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T05:26:02.278 INFO:teuthology.orchestra.run.smithi023.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-24T05:26:02.278 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl is present 2024-03-24T05:26:02.278 INFO:teuthology.orchestra.run.smithi023.stdout:lvcreate is present 2024-03-24T05:26:02.284 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T05:26:02.284 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T05:26:02.290 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T05:26:02.290 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-24T05:26:02.296 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout enabled 2024-03-24T05:26:02.302 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout active 2024-03-24T05:26:02.302 INFO:teuthology.orchestra.run.smithi023.stdout:Unit chronyd.service is enabled and running 2024-03-24T05:26:02.302 INFO:teuthology.orchestra.run.smithi023.stdout:Host looks OK 2024-03-24T05:26:02.302 INFO:teuthology.orchestra.run.smithi023.stdout:Cluster fsid: e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:26:02.302 INFO:teuthology.orchestra.run.smithi023.stdout:Acquiring lock 140306407756848 on /run/cephadm/e2aa55c2-e99e-11ee-95cd-87774f69a715.lock 2024-03-24T05:26:02.302 INFO:teuthology.orchestra.run.smithi023.stdout:Lock 140306407756848 acquired on /run/cephadm/e2aa55c2-e99e-11ee-95cd-87774f69a715.lock 2024-03-24T05:26:02.303 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying IP 172.21.15.23 port 3300 ... 2024-03-24T05:26:02.303 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying IP 172.21.15.23 port 6789 ... 2024-03-24T05:26:02.303 INFO:teuthology.orchestra.run.smithi023.stdout:Base mon IP(s) is [172.21.15.23:3300, 172.21.15.23:6789], mon addrv is [v2:172.21.15.23:3300,v1:172.21.15.23:6789] 2024-03-24T05:26:02.306 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.23 metric 100 2024-03-24T05:26:02.306 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.23 metric 100 2024-03-24T05:26:02.308 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-24T05:26:02.308 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-24T05:26:02.309 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-24T05:26:02.311 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-24T05:26:02.311 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-24T05:26:02.311 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T05:26:02.311 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-24T05:26:02.311 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe59:ce42/64 scope link noprefixroute 2024-03-24T05:26:02.311 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T05:26:02.312 INFO:teuthology.orchestra.run.smithi023.stdout:Mon IP `172.21.15.23` is in CIDR network `172.21.0.0/20` 2024-03-24T05:26:02.312 INFO:teuthology.orchestra.run.smithi023.stdout:Mon IP `172.21.15.23` is in CIDR network `172.21.0.0/20` 2024-03-24T05:26:02.312 INFO:teuthology.orchestra.run.smithi023.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-24T05:26:02.312 INFO:teuthology.orchestra.run.smithi023.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-24T05:26:02.312 INFO:teuthology.orchestra.run.smithi023.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T05:26:02.735 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d 2024-03-24T05:26:02.735 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T05:26:02.736 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Getting image source signatures 2024-03-24T05:26:02.736 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Copying blob sha256:989de1c398e87dbcb4906791a352373dc0e0415d619bc5b36f1f1823df929cb1 2024-03-24T05:26:02.736 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-24T05:26:02.736 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Copying config sha256:4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d 2024-03-24T05:26:02.736 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-24T05:26:03.672 INFO:teuthology.orchestra.run.smithi023.stdout:ceph: stdout ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev) 2024-03-24T05:26:03.672 INFO:teuthology.orchestra.run.smithi023.stdout:Ceph version: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev) 2024-03-24T05:26:03.672 INFO:teuthology.orchestra.run.smithi023.stdout:Extracting ceph user uid/gid from container image... 2024-03-24T05:26:04.677 INFO:teuthology.orchestra.run.smithi023.stdout:stat: stdout 167 167 2024-03-24T05:26:04.677 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial keys... 2024-03-24T05:26:05.766 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQBtuf9lJVrUARAAhgbWFMTBF907pC+Jj6SGHw== 2024-03-24T05:26:06.612 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQBuuf9ltVCYBRAARBAaRXbwueJTFMlhH6Th+g== 2024-03-24T05:26:07.417 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQBuuf9lgC4TNxAANI+8wnIWHgN44OAgxmPkPA== 2024-03-24T05:26:07.417 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial monmap... 2024-03-24T05:26:08.290 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T05:26:08.291 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-24T05:26:08.291 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:26:08.291 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T05:26:08.291 INFO:teuthology.orchestra.run.smithi023.stdout:monmaptool for a [v2:172.21.15.23:3300,v1:172.21.15.23:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T05:26:08.291 INFO:teuthology.orchestra.run.smithi023.stdout:setting min_mon_release = quincy 2024-03-24T05:26:08.291 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: set fsid to e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:26:08.291 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T05:26:08.291 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:26:08.292 INFO:teuthology.orchestra.run.smithi023.stdout:Creating mon... 2024-03-24T05:26:09.270 INFO:teuthology.orchestra.run.smithi023.stdout:create mon.a on 2024-03-24T05:26:09.645 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-24T05:26:09.823 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715.target. 2024-03-24T05:26:09.823 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715.target. 2024-03-24T05:26:10.073 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mon.a 2024-03-24T05:26:10.073 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-24T05:26:10.244 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715.target.wants/ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@.service. 2024-03-24T05:26:10.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:10 smithi023 podman[25344]: 2024-03-24 05:26:10.496485746 +0000 UTC m=+0.126819752 container create 8f660e16dde77f9aed40cb3491cb341f711f99da203a4ebd431b94f8293567fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-24T05:26:10.735 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T05:26:10.735 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-24T05:26:10.742 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T05:26:10.742 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-24T05:26:10.742 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-24T05:26:10.742 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T05:26:10.743 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mon to start... 2024-03-24T05:26:10.743 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mon... 2024-03-24T05:26:10.954 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:10 smithi023 podman[25344]: 2024-03-24 05:26:10.65708202 +0000 UTC m=+0.287415984 container init 8f660e16dde77f9aed40cb3491cb341f711f99da203a4ebd431b94f8293567fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-a, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-24T05:26:10.954 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:10 smithi023 podman[25344]: 2024-03-24 05:26:10.662205459 +0000 UTC m=+0.292539425 container start 8f660e16dde77f9aed40cb3491cb341f711f99da203a4ebd431b94f8293567fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-24T05:26:10.954 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:10 smithi023 bash[25344]: 8f660e16dde77f9aed40cb3491cb341f711f99da203a4ebd431b94f8293567fb 2024-03-24T05:26:10.955 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:10 smithi023 systemd[1]: Started Ceph mon.a for e2aa55c2-e99e-11ee-95cd-87774f69a715. 2024-03-24T05:26:10.955 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:10 smithi023 ceph-mon[25382]: mkfs e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:26:10.955 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:10 smithi023 ceph-mon[25382]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T05:26:11.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout cluster: 2024-03-24T05:26:11.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout id: e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:26:11.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-24T05:26:11.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T05:26:11.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout services: 2024-03-24T05:26:11.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.537692s) 2024-03-24T05:26:11.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-24T05:26:11.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-24T05:26:11.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T05:26:11.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout data: 2024-03-24T05:26:11.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-24T05:26:11.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-24T05:26:11.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-24T05:26:11.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout pgs: 2024-03-24T05:26:11.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T05:26:11.760 INFO:teuthology.orchestra.run.smithi023.stdout:mon is available 2024-03-24T05:26:11.760 INFO:teuthology.orchestra.run.smithi023.stdout:Assimilating anything we can from ceph.conf... 2024-03-24T05:26:11.998 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:11 smithi023 ceph-mon[25382]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T05:26:11.999 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:11 smithi023 ceph-mon[25382]: monmap e1: 1 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T05:26:12.000 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:11 smithi023 ceph-mon[25382]: fsmap 2024-03-24T05:26:12.000 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:11 smithi023 ceph-mon[25382]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T05:26:12.000 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:11 smithi023 ceph-mon[25382]: mgrmap e1: no daemons active 2024-03-24T05:26:12.000 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:11 smithi023 ceph-mon[25382]: from='client.? 172.21.15.23:0/1785749638' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-24T05:26:12.856 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T05:26:12.856 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T05:26:12.856 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout fsid = e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:26:12.856 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.23:3300,v1:172.21.15.23:6789] 2024-03-24T05:26:12.856 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T05:26:12.857 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T05:26:12.857 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T05:26:12.857 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T05:26:12.857 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T05:26:12.857 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T05:26:12.857 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = False 2024-03-24T05:26:12.857 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T05:26:12.857 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T05:26:12.857 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T05:26:12.857 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T05:26:12.857 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T05:26:12.858 INFO:teuthology.orchestra.run.smithi023.stdout:Generating new minimal ceph.conf... 2024-03-24T05:26:12.978 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:12 smithi023 ceph-mon[25382]: from='client.? 172.21.15.23:0/1015261445' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T05:26:12.978 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:12 smithi023 ceph-mon[25382]: from='client.? 172.21.15.23:0/1015261445' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-24T05:26:14.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:13 smithi023 ceph-mon[25382]: from='client.? 172.21.15.23:0/692956748' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:26:19.196 INFO:teuthology.orchestra.run.smithi023.stdout:Restarting the monitor... 2024-03-24T05:26:19.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:19 smithi023 systemd[1]: Stopping Ceph mon.a for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:26:20.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:19 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-a[25359]: 2024-03-24T05:26:19.633+0000 7f855253c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T05:26:20.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:19 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-a[25359]: 2024-03-24T05:26:19.633+0000 7f855253c700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T05:26:20.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:19 smithi023 podman[25715]: 2024-03-24 05:26:19.839376353 +0000 UTC m=+0.472947144 container died 8f660e16dde77f9aed40cb3491cb341f711f99da203a4ebd431b94f8293567fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-24T05:26:20.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 podman[25715]: 2024-03-24 05:26:20.17288872 +0000 UTC m=+0.806459518 container cleanup 8f660e16dde77f9aed40cb3491cb341f711f99da203a4ebd431b94f8293567fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-a, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-24T05:26:20.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 bash[25715]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-a 2024-03-24T05:26:20.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 podman[25734]: 2024-03-24 05:26:20.334788884 +0000 UTC m=+0.493575565 container remove 8f660e16dde77f9aed40cb3491cb341f711f99da203a4ebd431b94f8293567fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-a, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-24T05:26:20.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 systemd[1]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-24T05:26:20.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 systemd[1]: Stopped Ceph mon.a for e2aa55c2-e99e-11ee-95cd-87774f69a715. 2024-03-24T05:26:20.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 systemd[1]: Starting Ceph mon.a for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:26:20.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 podman[25831]: 2024-03-24 05:26:20.665537442 +0000 UTC m=+0.017349812 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T05:26:20.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 podman[25831]: 2024-03-24 05:26:20.766348913 +0000 UTC m=+0.118161278 container create 8e5cbfa90e96dc6d5e9350efa4ff75eece07fa707c50e1a69e98038ec8578f97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-a, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-24T05:26:20.996 INFO:teuthology.orchestra.run.smithi023.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-24T05:26:21.241 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 podman[25831]: 2024-03-24 05:26:20.9184516 +0000 UTC m=+0.270263968 container init 8e5cbfa90e96dc6d5e9350efa4ff75eece07fa707c50e1a69e98038ec8578f97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-a, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-24T05:26:21.241 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 podman[25831]: 2024-03-24 05:26:20.923581285 +0000 UTC m=+0.275393653 container start 8e5cbfa90e96dc6d5e9350efa4ff75eece07fa707c50e1a69e98038ec8578f97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T05:26:21.241 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T05:26:21.241 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 7 2024-03-24T05:26:21.241 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: pidfile_write: ignore empty --pid-file 2024-03-24T05:26:21.241 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: load: jerasure load: lrc 2024-03-24T05:26:21.241 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: RocksDB version: 7.9.2 2024-03-24T05:26:21.241 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Git sha 0 2024-03-24T05:26:21.241 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T05:26:21.241 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: DB SUMMARY 2024-03-24T05:26:21.242 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: DB Session ID: 2KAW93H3RB3WDTPNERVO 2024-03-24T05:26:21.242 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: CURRENT file: CURRENT 2024-03-24T05:26:21.242 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T05:26:21.242 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-24T05:26:21.242 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-24T05:26:21.242 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84625 ; 2024-03-24T05:26:21.242 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.error_if_exists: 0 2024-03-24T05:26:21.242 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.create_if_missing: 0 2024-03-24T05:26:21.242 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.paranoid_checks: 1 2024-03-24T05:26:21.242 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T05:26:21.242 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T05:26:21.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T05:26:21.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.env: 0x55594fa2fae0 2024-03-24T05:26:21.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T05:26:21.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.info_log: 0x55595184f8c0 2024-03-24T05:26:21.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T05:26:21.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.statistics: (nil) 2024-03-24T05:26:21.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.use_fsync: 0 2024-03-24T05:26:21.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_log_file_size: 0 2024-03-24T05:26:21.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T05:26:21.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T05:26:21.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T05:26:21.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T05:26:21.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.allow_fallocate: 1 2024-03-24T05:26:21.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T05:26:21.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T05:26:21.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.use_direct_reads: 0 2024-03-24T05:26:21.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T05:26:21.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T05:26:21.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.db_log_dir: 2024-03-24T05:26:21.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.wal_dir: 2024-03-24T05:26:21.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T05:26:21.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T05:26:21.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T05:26:21.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T05:26:21.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T05:26:21.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T05:26:21.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T05:26:21.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T05:26:21.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.write_buffer_manager: 0x555951823400 2024-03-24T05:26:21.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T05:26:21.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T05:26:21.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T05:26:21.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T05:26:21.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T05:26:21.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T05:26:21.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T05:26:21.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T05:26:21.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.unordered_write: 0 2024-03-24T05:26:21.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T05:26:21.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T05:26:21.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T05:26:21.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T05:26:21.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.row_cache: None 2024-03-24T05:26:21.246 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.wal_filter: None 2024-03-24T05:26:21.246 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T05:26:21.247 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T05:26:21.247 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.two_write_queues: 0 2024-03-24T05:26:21.247 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T05:26:21.247 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.wal_compression: 0 2024-03-24T05:26:21.247 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.atomic_flush: 0 2024-03-24T05:26:21.247 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T05:26:21.247 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T05:26:21.247 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T05:26:21.247 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.log_readahead_size: 0 2024-03-24T05:26:21.247 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T05:26:21.248 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T05:26:21.248 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T05:26:21.248 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T05:26:21.248 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T05:26:21.248 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T05:26:21.248 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T05:26:21.248 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_background_jobs: 2 2024-03-24T05:26:21.248 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_background_compactions: -1 2024-03-24T05:26:21.248 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_subcompactions: 1 2024-03-24T05:26:21.248 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T05:26:21.248 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T05:26:21.248 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T05:26:21.248 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T05:26:21.248 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T05:26:21.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T05:26:21.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T05:26:21.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T05:26:21.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_open_files: -1 2024-03-24T05:26:21.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T05:26:21.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T05:26:21.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T05:26:21.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T05:26:21.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_background_flushes: -1 2024-03-24T05:26:21.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Compression algorithms supported: 2024-03-24T05:26:21.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T05:26:21.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: kZSTD supported: 0 2024-03-24T05:26:21.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: kXpressCompression supported: 0 2024-03-24T05:26:21.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T05:26:21.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: kZlibCompression supported: 1 2024-03-24T05:26:21.250 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: kSnappyCompression supported: 1 2024-03-24T05:26:21.250 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: kLZ4Compression supported: 1 2024-03-24T05:26:21.250 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: kBZip2Compression supported: 0 2024-03-24T05:26:21.250 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T05:26:21.250 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T05:26:21.250 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-24T05:26:21.250 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T05:26:21.250 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T05:26:21.250 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.merge_operator: 2024-03-24T05:26:21.250 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compaction_filter: None 2024-03-24T05:26:21.250 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T05:26:21.250 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T05:26:21.250 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T05:26:21.250 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T05:26:21.251 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55595184f9c0) 2024-03-24T05:26:21.251 INFO:journalctl@ceph.mon.a.smithi023.stdout: cache_index_and_filter_blocks: 1 2024-03-24T05:26:21.251 INFO:journalctl@ceph.mon.a.smithi023.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T05:26:21.251 INFO:journalctl@ceph.mon.a.smithi023.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T05:26:21.251 INFO:journalctl@ceph.mon.a.smithi023.stdout: pin_top_level_index_and_filter: 1 2024-03-24T05:26:21.251 INFO:journalctl@ceph.mon.a.smithi023.stdout: index_type: 0 2024-03-24T05:26:21.251 INFO:journalctl@ceph.mon.a.smithi023.stdout: data_block_index_type: 0 2024-03-24T05:26:21.251 INFO:journalctl@ceph.mon.a.smithi023.stdout: index_shortening: 1 2024-03-24T05:26:21.251 INFO:journalctl@ceph.mon.a.smithi023.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T05:26:21.251 INFO:journalctl@ceph.mon.a.smithi023.stdout: checksum: 4 2024-03-24T05:26:21.251 INFO:journalctl@ceph.mon.a.smithi023.stdout: no_block_cache: 0 2024-03-24T05:26:21.251 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache: 0x555951853090 2024-03-24T05:26:21.251 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache_name: BinnedLRUCache 2024-03-24T05:26:21.251 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache_options: 2024-03-24T05:26:21.251 INFO:journalctl@ceph.mon.a.smithi023.stdout: capacity : 536870912 2024-03-24T05:26:21.252 INFO:journalctl@ceph.mon.a.smithi023.stdout: num_shard_bits : 4 2024-03-24T05:26:21.252 INFO:journalctl@ceph.mon.a.smithi023.stdout: strict_capacity_limit : 0 2024-03-24T05:26:21.252 INFO:journalctl@ceph.mon.a.smithi023.stdout: high_pri_pool_ratio: 0.000 2024-03-24T05:26:21.252 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache_compressed: (nil) 2024-03-24T05:26:21.252 INFO:journalctl@ceph.mon.a.smithi023.stdout: persistent_cache: (nil) 2024-03-24T05:26:21.252 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_size: 4096 2024-03-24T05:26:21.252 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_size_deviation: 10 2024-03-24T05:26:21.252 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_restart_interval: 16 2024-03-24T05:26:21.252 INFO:journalctl@ceph.mon.a.smithi023.stdout: index_block_restart_interval: 1 2024-03-24T05:26:21.252 INFO:journalctl@ceph.mon.a.smithi023.stdout: metadata_block_size: 4096 2024-03-24T05:26:21.252 INFO:journalctl@ceph.mon.a.smithi023.stdout: partition_filters: 0 2024-03-24T05:26:21.252 INFO:journalctl@ceph.mon.a.smithi023.stdout: use_delta_encoding: 1 2024-03-24T05:26:21.252 INFO:journalctl@ceph.mon.a.smithi023.stdout: filter_policy: bloomfilter 2024-03-24T05:26:21.252 INFO:journalctl@ceph.mon.a.smithi023.stdout: whole_key_filtering: 1 2024-03-24T05:26:21.252 INFO:journalctl@ceph.mon.a.smithi023.stdout: verify_compression: 0 2024-03-24T05:26:21.253 INFO:journalctl@ceph.mon.a.smithi023.stdout: read_amp_bytes_per_bit: 0 2024-03-24T05:26:21.253 INFO:journalctl@ceph.mon.a.smithi023.stdout: format_version: 5 2024-03-24T05:26:21.253 INFO:journalctl@ceph.mon.a.smithi023.stdout: enable_index_compression: 1 2024-03-24T05:26:21.253 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_align: 0 2024-03-24T05:26:21.253 INFO:journalctl@ceph.mon.a.smithi023.stdout: max_auto_readahead_size: 262144 2024-03-24T05:26:21.253 INFO:journalctl@ceph.mon.a.smithi023.stdout: prepopulate_block_cache: 0 2024-03-24T05:26:21.253 INFO:journalctl@ceph.mon.a.smithi023.stdout: initial_auto_readahead_size: 8192 2024-03-24T05:26:21.253 INFO:journalctl@ceph.mon.a.smithi023.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T05:26:21.253 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T05:26:21.253 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T05:26:21.253 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compression: NoCompression 2024-03-24T05:26:21.253 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T05:26:21.253 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T05:26:21.253 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T05:26:21.253 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.num_levels: 7 2024-03-24T05:26:21.254 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T05:26:21.254 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T05:26:21.254 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T05:26:21.254 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T05:26:21.254 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T05:26:21.254 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T05:26:21.254 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T05:26:21.254 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T05:26:21.254 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T05:26:21.254 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T05:26:21.254 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T05:26:21.254 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T05:26:21.254 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T05:26:21.254 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T05:26:21.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T05:26:21.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T05:26:21.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T05:26:21.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T05:26:21.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T05:26:21.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T05:26:21.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T05:26:21.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T05:26:21.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T05:26:21.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T05:26:21.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T05:26:21.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T05:26:21.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T05:26:21.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T05:26:21.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T05:26:21.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T05:26:21.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T05:26:21.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T05:26:21.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T05:26:21.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T05:26:21.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T05:26:21.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T05:26:21.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T05:26:21.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T05:26:21.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T05:26:21.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T05:26:21.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T05:26:21.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T05:26:21.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T05:26:21.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T05:26:21.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T05:26:21.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T05:26:21.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T05:26:21.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T05:26:21.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T05:26:21.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T05:26:21.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T05:26:21.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T05:26:21.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T05:26:21.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.table_properties_collectors: 2024-03-24T05:26:21.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.inplace_update_support: 0 2024-03-24T05:26:21.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T05:26:21.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T05:26:21.258 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T05:26:21.258 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T05:26:21.258 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.bloom_locality: 0 2024-03-24T05:26:21.258 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.max_successive_merges: 0 2024-03-24T05:26:21.258 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T05:26:21.258 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T05:26:21.258 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T05:26:21.258 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T05:26:21.258 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.ttl: 2592000 2024-03-24T05:26:21.258 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T05:26:21.258 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T05:26:21.258 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T05:26:21.258 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.enable_blob_files: false 2024-03-24T05:26:21.259 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.min_blob_size: 0 2024-03-24T05:26:21.259 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T05:26:21.259 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T05:26:21.259 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T05:26:21.259 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T05:26:21.259 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T05:26:21.259 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T05:26:21.259 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T05:26:21.259 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T05:26:21.259 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-24T05:26:21.259 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-24T05:26:21.259 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c0bcf039-90b5-4d6a-a65c-5de7996a646a 2024-03-24T05:26:21.259 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711257980952765, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-24T05:26:21.259 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-24T05:26:21.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711257980954198, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81250, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79437, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73746, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711257980, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c0bcf039-90b5-4d6a-a65c-5de7996a646a", "db_session_id": "2KAW93H3RB3WDTPNERVO", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-24T05:26:21.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711257980954306, "job": 1, "event": "recovery_finished"} 2024-03-24T05:26:21.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-24T05:26:21.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T05:26:21.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555951944000 2024-03-24T05:26:21.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: DB pointer 0x55595192e000 2024-03-24T05:26:21.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T05:26:21.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T05:26:21.260 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** DB Stats ** 2024-03-24T05:26:21.260 INFO:journalctl@ceph.mon.a.smithi023.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T05:26:21.260 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T05:26:21.260 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T05:26:21.261 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T05:26:21.261 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T05:26:21.261 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T05:26:21.261 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T05:26:21.261 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-24T05:26:21.261 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** Compaction Stats [default] ** 2024-03-24T05:26:21.261 INFO:journalctl@ceph.mon.a.smithi023.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T05:26:21.261 INFO:journalctl@ceph.mon.a.smithi023.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T05:26:21.262 INFO:journalctl@ceph.mon.a.smithi023.stdout: L0 2/0 81.18 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 81.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T05:26:21.262 INFO:journalctl@ceph.mon.a.smithi023.stdout: Sum 2/0 81.18 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 81.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T05:26:21.262 INFO:journalctl@ceph.mon.a.smithi023.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 81.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T05:26:21.262 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-24T05:26:21.262 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** Compaction Stats [default] ** 2024-03-24T05:26:21.262 INFO:journalctl@ceph.mon.a.smithi023.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T05:26:21.262 INFO:journalctl@ceph.mon.a.smithi023.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T05:26:21.262 INFO:journalctl@ceph.mon.a.smithi023.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 81.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T05:26:21.262 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-24T05:26:21.262 INFO:journalctl@ceph.mon.a.smithi023.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T05:26:21.262 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-24T05:26:21.263 INFO:journalctl@ceph.mon.a.smithi023.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T05:26:21.263 INFO:journalctl@ceph.mon.a.smithi023.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T05:26:21.263 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T05:26:21.263 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T05:26:21.263 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T05:26:21.263 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T05:26:21.263 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative compaction: 0.00 GB write, 12.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T05:26:21.263 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval compaction: 0.00 GB write, 12.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T05:26:21.263 INFO:journalctl@ceph.mon.a.smithi023.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T05:26:21.263 INFO:journalctl@ceph.mon.a.smithi023.stdout: Block cache BinnedLRUCache@0x555951853090#7 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-03-24T05:26:21.263 INFO:journalctl@ceph.mon.a.smithi023.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-24T05:26:21.263 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-24T05:26:21.263 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T05:26:21.263 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: starting mon.a rank 0 at public addrs [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] at bind addrs [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:26:21.263 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: mon.a@-1(???) e1 preinit fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:26:21.264 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: mon.a@-1(???).mds e1 new map 2024-03-24T05:26:21.264 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: mon.a@-1(???).mds e1 print_map 2024-03-24T05:26:21.264 INFO:journalctl@ceph.mon.a.smithi023.stdout: e1 2024-03-24T05:26:21.264 INFO:journalctl@ceph.mon.a.smithi023.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T05:26:21.264 INFO:journalctl@ceph.mon.a.smithi023.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T05:26:21.264 INFO:journalctl@ceph.mon.a.smithi023.stdout: legacy client fscid: -1 2024-03-24T05:26:21.264 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-24T05:26:21.264 INFO:journalctl@ceph.mon.a.smithi023.stdout: No filesystems configured 2024-03-24T05:26:21.264 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T05:26:21.264 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T05:26:21.264 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T05:26:21.264 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T05:26:21.265 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 ceph-mon[25869]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-24T05:26:21.265 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 bash[25831]: 8e5cbfa90e96dc6d5e9350efa4ff75eece07fa707c50e1a69e98038ec8578f97 2024-03-24T05:26:21.265 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:20 smithi023 systemd[1]: Started Ceph mon.a for e2aa55c2-e99e-11ee-95cd-87774f69a715. 2024-03-24T05:26:21.265 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:21 smithi023 ceph-mon[25869]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T05:26:21.265 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:21 smithi023 ceph-mon[25869]: monmap e1: 1 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T05:26:21.265 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:21 smithi023 ceph-mon[25869]: fsmap 2024-03-24T05:26:21.265 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:21 smithi023 ceph-mon[25869]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T05:26:21.265 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:21 smithi023 ceph-mon[25869]: mgrmap e1: no daemons active 2024-03-24T05:26:22.126 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-24T05:26:22.147 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-24T05:26:22.148 INFO:teuthology.orchestra.run.smithi023.stdout:Creating mgr... 2024-03-24T05:26:22.148 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-24T05:26:22.148 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-24T05:26:22.417 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mgr.y 2024-03-24T05:26:22.417 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mgr.y.service: Unit ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mgr.y.service not loaded. 2024-03-24T05:26:22.567 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715.target.wants/ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@.service. 2024-03-24T05:26:22.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:22 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/3020696471' entity='client.admin' 2024-03-24T05:26:22.731 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:22 smithi023 systemd[1]: Starting Ceph mgr.y for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:26:23.002 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:22 smithi023 podman[26113]: 2024-03-24 05:26:22.707367446 +0000 UTC m=+0.017665103 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T05:26:23.002 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:22 smithi023 podman[26113]: 2024-03-24 05:26:22.818075883 +0000 UTC m=+0.128373527 container create 877f2bae6dcbd50a250036be842e1b30a933533a0f7102b008752e10180bd804 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-24T05:26:23.002 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:23 smithi023 podman[26113]: 2024-03-24 05:26:23.003451207 +0000 UTC m=+0.313748860 container init 877f2bae6dcbd50a250036be842e1b30a933533a0f7102b008752e10180bd804 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-24T05:26:23.076 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T05:26:23.076 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-24T05:26:23.084 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T05:26:23.084 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-24T05:26:23.084 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-24T05:26:23.084 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T05:26:23.093 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T05:26:23.093 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-24T05:26:23.100 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T05:26:23.100 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-24T05:26:23.100 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-24T05:26:23.100 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-24T05:26:23.100 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr to start... 2024-03-24T05:26:23.100 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr... 2024-03-24T05:26:23.329 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:23 smithi023 podman[26113]: 2024-03-24 05:26:23.008500632 +0000 UTC m=+0.318798276 container start 877f2bae6dcbd50a250036be842e1b30a933533a0f7102b008752e10180bd804 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y, RELEASE=squid-db0330b, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-24T05:26:23.329 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:23 smithi023 bash[26113]: 877f2bae6dcbd50a250036be842e1b30a933533a0f7102b008752e10180bd804 2024-03-24T05:26:23.330 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:23 smithi023 systemd[1]: Started Ceph mgr.y for e2aa55c2-e99e-11ee-95cd-87774f69a715. 2024-03-24T05:26:23.949 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:23 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/2581171118' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T05:26:23.949 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:23 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:23.594+0000 7fe3bc556200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T05:26:23.949 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:23 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:23.694+0000 7fe3bc556200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T05:26:24.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T05:26:24.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-24T05:26:24.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "e2aa55c2-e99e-11ee-95cd-87774f69a715", 2024-03-24T05:26:24.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T05:26:24.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T05:26:24.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T05:26:24.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T05:26:24.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:24.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T05:26:24.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T05:26:24.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-24T05:26:24.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T05:26:24.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T05:26:24.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T05:26:24.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T05:26:24.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-24T05:26:24.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T05:26:24.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T05:26:24.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T05:26:24.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T05:26:24.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:24.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T05:26:24.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T05:26:24.232 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T05:26:24.232 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T05:26:24.232 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T05:26:24.232 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T05:26:24.232 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T05:26:24.232 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T05:26:24.235 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:24.236 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T05:26:24.236 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T05:26:24.236 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T05:26:24.236 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T05:26:24.236 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T05:26:24.236 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T05:26:24.236 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T05:26:24.236 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T05:26:24.236 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T05:26:24.236 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:24.236 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T05:26:24.236 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T05:26:24.236 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T05:26:24.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T05:26:24.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:24.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T05:26:24.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T05:26:24.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T05:26:24.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T05:26:24.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T05:26:24.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T05:26:24.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T05:26:24.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T05:26:24.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T05:26:24.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:24.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T05:26:24.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T05:26:24.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T05:26:10.742957+0000", 2024-03-24T05:26:24.238 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T05:26:24.238 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:24.238 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T05:26:24.238 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-24T05:26:24.238 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (1/15)... 2024-03-24T05:26:24.579 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:24 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:24.363+0000 7fe3bc556200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T05:26:24.579 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:24 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:24.458+0000 7fe3bc556200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T05:26:25.066 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:24 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:24.601+0000 7fe3bc556200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T05:26:25.066 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:24 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:24.699+0000 7fe3bc556200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T05:26:25.066 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:24 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:24.784+0000 7fe3bc556200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T05:26:25.066 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:25 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:25.065+0000 7fe3bc556200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T05:26:25.329 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:25 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:25.179+0000 7fe3bc556200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T05:26:25.329 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:25 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:25.269+0000 7fe3bc556200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T05:26:25.829 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:25 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:25.367+0000 7fe3bc556200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T05:26:25.830 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:25 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:25.547+0000 7fe3bc556200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T05:26:26.690 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:26 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:26.556+0000 7fe3bc556200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T05:26:26.691 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:26 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:26.690+0000 7fe3bc556200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T05:26:27.262 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:27 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:27.047+0000 7fe3bc556200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T05:26:27.262 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:27 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:27.144+0000 7fe3bc556200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T05:26:27.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:27 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/801878419' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T05:26:27.541 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:27 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:27.349+0000 7fe3bc556200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T05:26:27.541 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:27 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:27.540+0000 7fe3bc556200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T05:26:27.949 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T05:26:27.949 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-24T05:26:27.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "e2aa55c2-e99e-11ee-95cd-87774f69a715", 2024-03-24T05:26:27.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T05:26:27.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T05:26:27.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T05:26:27.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T05:26:27.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:27.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T05:26:27.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T05:26:27.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-24T05:26:27.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T05:26:27.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T05:26:27.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T05:26:27.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T05:26:27.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-24T05:26:27.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T05:26:27.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T05:26:27.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T05:26:27.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T05:26:27.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:27.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T05:26:27.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T05:26:27.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T05:26:27.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T05:26:27.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T05:26:27.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T05:26:27.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T05:26:27.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T05:26:27.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:27.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T05:26:27.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T05:26:27.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T05:26:27.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T05:26:27.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T05:26:27.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T05:26:27.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T05:26:27.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T05:26:27.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T05:26:27.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:27.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T05:26:27.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T05:26:27.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T05:26:27.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T05:26:27.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:27.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T05:26:27.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T05:26:27.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T05:26:27.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T05:26:27.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T05:26:27.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T05:26:27.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T05:26:27.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T05:26:27.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T05:26:27.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:27.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T05:26:27.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T05:26:27.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T05:26:10.742957+0000", 2024-03-24T05:26:27.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T05:26:27.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:27.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T05:26:27.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-24T05:26:27.956 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (2/15)... 2024-03-24T05:26:28.102 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:27 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:27.840+0000 7fe3bc556200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T05:26:28.103 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:28 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:28.103+0000 7fe3bc556200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T05:26:28.579 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:28 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:28.347+0000 7fe3bc556200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T05:26:29.580 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:29 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:29.176+0000 7fe3bc556200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T05:26:30.067 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:29 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:29.790+0000 7fe3bc556200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T05:26:30.067 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:29 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:29.890+0000 7fe3bc556200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T05:26:30.329 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:30 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:30.118+0000 7fe3bc556200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T05:26:30.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:30 smithi023 ceph-mon[25869]: Activating manager daemon y 2024-03-24T05:26:30.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:30 smithi023 ceph-mon[25869]: mgrmap e2: y(active, starting, since 0.00309374s) 2024-03-24T05:26:30.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:30 smithi023 ceph-mon[25869]: from='mgr.14100 172.21.15.23:0/2014931928' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T05:26:30.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:30 smithi023 ceph-mon[25869]: from='mgr.14100 172.21.15.23:0/2014931928' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T05:26:30.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:30 smithi023 ceph-mon[25869]: from='mgr.14100 172.21.15.23:0/2014931928' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T05:26:30.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:30 smithi023 ceph-mon[25869]: from='mgr.14100 172.21.15.23:0/2014931928' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T05:26:30.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:30 smithi023 ceph-mon[25869]: from='mgr.14100 172.21.15.23:0/2014931928' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T05:26:30.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:30 smithi023 ceph-mon[25869]: Manager daemon y is now available 2024-03-24T05:26:30.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:30 smithi023 ceph-mon[25869]: from='mgr.14100 172.21.15.23:0/2014931928' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T05:26:30.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:30 smithi023 ceph-mon[25869]: from='mgr.14100 172.21.15.23:0/2014931928' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T05:26:30.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:30 smithi023 ceph-mon[25869]: from='mgr.14100 172.21.15.23:0/2014931928' entity='mgr.y' 2024-03-24T05:26:30.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:30 smithi023 ceph-mon[25869]: from='mgr.14100 172.21.15.23:0/2014931928' entity='mgr.y' 2024-03-24T05:26:30.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:30 smithi023 ceph-mon[25869]: from='mgr.14100 172.21.15.23:0/2014931928' entity='mgr.y' 2024-03-24T05:26:30.991 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T05:26:30.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-24T05:26:30.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "e2aa55c2-e99e-11ee-95cd-87774f69a715", 2024-03-24T05:26:30.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T05:26:30.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T05:26:30.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T05:26:30.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T05:26:30.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:30.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T05:26:30.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T05:26:30.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-24T05:26:30.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T05:26:30.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T05:26:30.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T05:26:30.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T05:26:30.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-24T05:26:30.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T05:26:30.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T05:26:30.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T05:26:30.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T05:26:30.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:30.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T05:26:30.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T05:26:30.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T05:26:30.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T05:26:30.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T05:26:30.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T05:26:30.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T05:26:30.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T05:26:30.997 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:30.997 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T05:26:30.997 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T05:26:30.997 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T05:26:30.997 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T05:26:30.997 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T05:26:30.997 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T05:26:30.997 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T05:26:30.997 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T05:26:30.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T05:26:30.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:30.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T05:26:30.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T05:26:30.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T05:26:30.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T05:26:30.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:30.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T05:26:30.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T05:26:30.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T05:26:30.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T05:26:30.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T05:26:30.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T05:26:30.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T05:26:30.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T05:26:30.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T05:26:30.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:30.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T05:26:30.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T05:26:30.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T05:26:10.742957+0000", 2024-03-24T05:26:30.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T05:26:30.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:30.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T05:26:30.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-24T05:26:30.999 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (3/15)... 2024-03-24T05:26:31.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:31 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/2419819427' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T05:26:31.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:31 smithi023 ceph-mon[25869]: mgrmap e3: y(active, since 1.00758s) 2024-03-24T05:26:33.372 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:33 smithi023 ceph-mon[25869]: mgrmap e4: y(active, since 2s) 2024-03-24T05:26:34.373 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T05:26:34.373 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-24T05:26:34.373 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "e2aa55c2-e99e-11ee-95cd-87774f69a715", 2024-03-24T05:26:34.373 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T05:26:34.374 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T05:26:34.374 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T05:26:34.374 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T05:26:34.374 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:34.374 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T05:26:34.374 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T05:26:34.374 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-24T05:26:34.374 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T05:26:34.374 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T05:26:34.374 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T05:26:34.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T05:26:34.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-24T05:26:34.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T05:26:34.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T05:26:34.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T05:26:34.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T05:26:34.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:34.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T05:26:34.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T05:26:34.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T05:26:34.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T05:26:34.376 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T05:26:34.376 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T05:26:34.380 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T05:26:34.380 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T05:26:34.381 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:34.381 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T05:26:34.381 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T05:26:34.381 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T05:26:34.381 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T05:26:34.381 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T05:26:34.381 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T05:26:34.381 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T05:26:34.381 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T05:26:34.381 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T05:26:34.382 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:34.382 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T05:26:34.382 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T05:26:34.382 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T05:26:34.382 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T05:26:34.382 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:34.382 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T05:26:34.382 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T05:26:34.382 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T05:26:34.382 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T05:26:34.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T05:26:34.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T05:26:34.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T05:26:34.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T05:26:34.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T05:26:34.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:34.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T05:26:34.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T05:26:34.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T05:26:10.742957+0000", 2024-03-24T05:26:34.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T05:26:34.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T05:26:34.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T05:26:34.384 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-24T05:26:34.384 INFO:teuthology.orchestra.run.smithi023.stdout:mgr is available 2024-03-24T05:26:34.476 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:34 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/1540529282' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T05:26:35.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:35 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/3319129487' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T05:26:35.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:35 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/3319129487' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-24T05:26:35.537 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T05:26:35.537 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T05:26:35.537 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout fsid = e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:26:35.537 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T05:26:35.537 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T05:26:35.537 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T05:26:35.538 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T05:26:35.538 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T05:26:35.538 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T05:26:35.538 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T05:26:35.538 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T05:26:35.538 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T05:26:35.538 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T05:26:35.538 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T05:26:35.538 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling cephadm module... 2024-03-24T05:26:36.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:36 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/4063045052' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-24T05:26:36.579 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:36 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: ignoring --setuser ceph since I am not root 2024-03-24T05:26:36.580 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:36 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: ignoring --setgroup ceph since I am not root 2024-03-24T05:26:37.179 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:36 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:36.897+0000 7f6ddc00e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T05:26:37.180 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:36 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:36.998+0000 7f6ddc00e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T05:26:37.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:37 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/4063045052' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-24T05:26:37.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:37 smithi023 ceph-mon[25869]: mgrmap e5: y(active, since 6s) 2024-03-24T05:26:37.944 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:37 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:37.662+0000 7f6ddc00e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T05:26:37.944 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:37 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:37.766+0000 7f6ddc00e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T05:26:37.944 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:37 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:37.912+0000 7f6ddc00e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T05:26:38.203 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:38 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:38.011+0000 7f6ddc00e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T05:26:38.203 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:38 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:38.096+0000 7f6ddc00e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T05:26:38.207 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-24T05:26:38.207 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-24T05:26:38.207 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T05:26:38.207 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-24T05:26:38.207 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T05:26:38.207 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-24T05:26:38.207 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for the mgr to restart... 2024-03-24T05:26:38.207 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr epoch 5... 2024-03-24T05:26:38.470 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:38 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/3635597022' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T05:26:38.470 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:38 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:38.371+0000 7f6ddc00e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T05:26:38.829 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:38 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:38.486+0000 7f6ddc00e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T05:26:38.829 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:38 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:38.575+0000 7f6ddc00e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T05:26:38.830 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:38 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:38.673+0000 7f6ddc00e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T05:26:39.329 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:38 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:38.853+0000 7f6ddc00e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T05:26:40.302 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:39 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:39.834+0000 7f6ddc00e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T05:26:40.302 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:39 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:39.967+0000 7f6ddc00e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T05:26:40.302 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:40 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:40.303+0000 7f6ddc00e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T05:26:40.579 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:40 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:40.395+0000 7f6ddc00e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T05:26:40.886 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:40 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:40.595+0000 7f6ddc00e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T05:26:40.886 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:40 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:40.784+0000 7f6ddc00e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T05:26:41.329 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:41 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:41.078+0000 7f6ddc00e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T05:26:41.829 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:41 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:41.339+0000 7f6ddc00e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T05:26:41.829 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:41 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:41.584+0000 7f6ddc00e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T05:26:42.829 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:42 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:42.406+0000 7f6ddc00e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T05:26:43.306 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:42 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:42.988+0000 7f6ddc00e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T05:26:43.306 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:43 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:43.087+0000 7f6ddc00e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T05:26:43.307 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:43 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:26:43.307+0000 7f6ddc00e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T05:26:43.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:43 smithi023 ceph-mon[25869]: Active manager daemon y restarted 2024-03-24T05:26:43.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:43 smithi023 ceph-mon[25869]: Activating manager daemon y 2024-03-24T05:26:43.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:43 smithi023 ceph-mon[25869]: osdmap e2: 0 total, 0 up, 0 in 2024-03-24T05:26:43.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:43 smithi023 ceph-mon[25869]: mgrmap e6: y(active, starting, since 0.00563684s) 2024-03-24T05:26:43.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:43 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T05:26:43.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:43 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T05:26:43.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:43 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T05:26:43.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:43 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T05:26:43.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:43 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T05:26:43.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:43 smithi023 ceph-mon[25869]: Manager daemon y is now available 2024-03-24T05:26:43.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:43 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' 2024-03-24T05:26:43.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:43 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' 2024-03-24T05:26:43.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:43 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:26:43.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:43 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:26:44.625 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:44 smithi023 ceph-mon[25869]: Found migration_current of "None". Setting to last migration. 2024-03-24T05:26:44.626 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:44 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:26:44.626 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:44 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T05:26:44.626 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:44 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T05:26:44.626 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:44 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' 2024-03-24T05:26:44.626 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:44 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' 2024-03-24T05:26:44.626 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:44 smithi023 ceph-mon[25869]: mgrmap e7: y(active, since 1.00558s) 2024-03-24T05:26:44.862 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-24T05:26:44.862 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-24T05:26:44.862 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T05:26:44.862 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-24T05:26:44.863 INFO:teuthology.orchestra.run.smithi023.stdout:mgr epoch 5 is available 2024-03-24T05:26:44.863 INFO:teuthology.orchestra.run.smithi023.stdout:Setting orchestrator backend to cephadm... 2024-03-24T05:26:45.564 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:45 smithi023 ceph-mon[25869]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T05:26:45.564 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:45 smithi023 ceph-mon[25869]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T05:26:46.636 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:46 smithi023 ceph-mon[25869]: [24/Mar/2024:05:26:45] ENGINE Bus STARTING 2024-03-24T05:26:46.636 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:46 smithi023 ceph-mon[25869]: mgrmap e8: y(active, since 2s) 2024-03-24T05:26:46.636 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:46 smithi023 ceph-mon[25869]: [24/Mar/2024:05:26:45] ENGINE Serving on https://172.21.15.23:7150 2024-03-24T05:26:46.636 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:46 smithi023 ceph-mon[25869]: [24/Mar/2024:05:26:45] ENGINE Serving on http://172.21.15.23:8765 2024-03-24T05:26:46.636 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:46 smithi023 ceph-mon[25869]: [24/Mar/2024:05:26:45] ENGINE Bus STARTED 2024-03-24T05:26:46.636 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:46 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:26:46.637 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:46 smithi023 ceph-mon[25869]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:26:46.637 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:46 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' 2024-03-24T05:26:46.637 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:46 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:26:47.605 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:47 smithi023 ceph-mon[25869]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:26:47.606 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-24T05:26:47.606 INFO:teuthology.orchestra.run.smithi023.stdout:Generating ssh key... 2024-03-24T05:26:48.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:48 smithi023 ceph-mon[25869]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:26:48.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:48 smithi023 ceph-mon[25869]: Generating ssh key... 2024-03-24T05:26:48.785 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:48 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: Generating public/private rsa key pair. 2024-03-24T05:26:48.785 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:48 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: Your identification has been saved in /tmp/tmphfn3vmia/key. 2024-03-24T05:26:48.785 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:48 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: Your public key has been saved in /tmp/tmphfn3vmia/key.pub. 2024-03-24T05:26:48.785 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:48 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: The key fingerprint is: 2024-03-24T05:26:48.785 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:48 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: SHA256:DehMqedoq6dm5nIgHAbB1tP2TSL4QbG1javQQ8Mti+8 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:26:48.785 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:48 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: The key's randomart image is: 2024-03-24T05:26:48.785 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:48 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: +---[RSA 3072]----+ 2024-03-24T05:26:48.786 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:48 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: |o.. +o.. | 2024-03-24T05:26:48.786 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:48 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: |.o +.=+++. | 2024-03-24T05:26:48.786 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:48 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: |o +*B+=. | 2024-03-24T05:26:48.786 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:48 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: | o +*+..+ | 2024-03-24T05:26:48.786 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:48 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: |o .o.++.S . | 2024-03-24T05:26:48.786 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:48 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: |o. o+o | 2024-03-24T05:26:48.786 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:48 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: |.. oo. | 2024-03-24T05:26:48.786 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:48 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: |. =.o. | 2024-03-24T05:26:48.786 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:48 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: | B++.E | 2024-03-24T05:26:48.786 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:26:48 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: +----[SHA256]-----+ 2024-03-24T05:26:49.690 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:49 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' 2024-03-24T05:26:49.690 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:49 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' 2024-03-24T05:26:50.250 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvJKTk74/y/amuZCo0pwTTDWxkyr9ZsDKXDk3UGFWuG0p4rvKVduZCjM3LknLU/IAxV/52nb0YnyImq+BrcrGgpOWh7hqDaISdjx2bMk6C3VSLgGTz4jYeB0uwiaCSUJ8+0uSshwWgW08E2N/4xX5x8o4q3zZwPF0rDRcCp59D/KSoTDZY/pxwfmGHy6p2EI2UwExUrdtYMzeQFEMJ8AyG5tv8weRhk17ZAIqUohKH6Z/8n0Mt/yppEUledBhFrayhEKVg1LPmihQn+uOrTdMcBxa0SAFI3MDdwuPALmPKCkM8XBW2qVqa/MgczaiGYwFVchd9uPjl9wUBKccUNnbo/LcrXEs4KxOOHqwFx4pu+XSKehow9Nrmey2G6LNgMrmnLF4fT5V55s+cRNOpJlxBdK2/BJRsmbE9TynlCuvQ0+jr7dNyHYKq+PtoGev5667m09bp+6Yrx1aKEck2jcyt2fWwybkBhmUndbWCkgllTdYt/vq83mwWx4f/1gDt6KE= ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:26:50.251 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-24T05:26:50.251 INFO:teuthology.orchestra.run.smithi023.stdout:Adding key to root@localhost authorized_keys... 2024-03-24T05:26:50.251 INFO:teuthology.orchestra.run.smithi023.stdout:Adding host smithi023... 2024-03-24T05:26:50.507 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:50 smithi023 ceph-mon[25869]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:26:51.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:51 smithi023 ceph-mon[25869]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi023", "addr": "172.21.15.23", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:26:52.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:52 smithi023 ceph-mon[25869]: Deploying cephadm binary to smithi023 2024-03-24T05:26:55.841 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Added host 'smithi023' with addr '172.21.15.23' 2024-03-24T05:26:55.841 INFO:teuthology.orchestra.run.smithi023.stdout:Deploying unmanaged mon service... 2024-03-24T05:26:56.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:56 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' 2024-03-24T05:26:56.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:56 smithi023 ceph-mon[25869]: Added host smithi023 2024-03-24T05:26:56.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:56 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:26:57.139 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-24T05:26:57.139 INFO:teuthology.orchestra.run.smithi023.stdout:Deploying unmanaged mgr service... 2024-03-24T05:26:57.788 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:57 smithi023 ceph-mon[25869]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:26:57.788 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:57 smithi023 ceph-mon[25869]: Saving service mon spec with placement count:5 2024-03-24T05:26:57.788 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:57 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' 2024-03-24T05:26:58.699 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-24T05:26:58.951 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:58 smithi023 ceph-mon[25869]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:26:58.951 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:58 smithi023 ceph-mon[25869]: Saving service mgr spec with placement count:2 2024-03-24T05:26:58.951 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:26:58 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' 2024-03-24T05:27:00.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:00 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' 2024-03-24T05:27:00.422 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:00 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/1414493447' entity='client.admin' 2024-03-24T05:27:00.422 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:00 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' 2024-03-24T05:27:01.774 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling the dashboard module... 2024-03-24T05:27:02.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:02 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/2630547719' entity='client.admin' 2024-03-24T05:27:03.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:03 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/1455004873' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-24T05:27:03.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:03 smithi023 ceph-mon[25869]: from='mgr.14120 172.21.15.23:0/3453179668' entity='mgr.y' 2024-03-24T05:27:03.330 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:03 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: ignoring --setuser ceph since I am not root 2024-03-24T05:27:03.330 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:03 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: ignoring --setgroup ceph since I am not root 2024-03-24T05:27:03.929 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:03 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:03.663+0000 7f8f14df3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T05:27:03.929 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:03 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:03.762+0000 7f8f14df3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T05:27:04.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:04 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/1455004873' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-24T05:27:04.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:04 smithi023 ceph-mon[25869]: mgrmap e9: y(active, since 19s) 2024-03-24T05:27:04.767 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:04 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:04.432+0000 7f8f14df3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T05:27:04.767 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:04 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:04.530+0000 7f8f14df3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T05:27:04.767 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:04 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:04.676+0000 7f8f14df3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T05:27:05.034 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:04 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:04.775+0000 7f8f14df3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T05:27:05.034 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:04 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:04.861+0000 7f8f14df3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T05:27:05.036 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-24T05:27:05.037 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-24T05:27:05.037 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T05:27:05.037 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-24T05:27:05.038 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T05:27:05.038 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-24T05:27:05.038 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for the mgr to restart... 2024-03-24T05:27:05.038 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr epoch 9... 2024-03-24T05:27:05.294 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:05 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/500977918' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T05:27:05.294 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:05 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:05.136+0000 7f8f14df3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T05:27:05.294 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:05 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:05.250+0000 7f8f14df3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T05:27:05.545 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:05 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:05.343+0000 7f8f14df3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T05:27:05.545 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:05 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:05.440+0000 7f8f14df3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T05:27:05.829 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:05 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:05.623+0000 7f8f14df3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T05:27:07.079 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:06 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:06.607+0000 7f8f14df3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T05:27:07.079 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:06 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:06.742+0000 7f8f14df3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T05:27:07.386 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:07 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:07.092+0000 7f8f14df3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T05:27:07.386 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:07 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:07.184+0000 7f8f14df3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T05:27:07.386 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:07 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:07.387+0000 7f8f14df3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T05:27:07.829 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:07 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:07.581+0000 7f8f14df3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T05:27:08.160 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:07 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:07.878+0000 7f8f14df3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T05:27:08.579 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:08 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:08.161+0000 7f8f14df3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T05:27:08.580 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:08 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:08.408+0000 7f8f14df3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T05:27:09.579 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:09 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:09.247+0000 7f8f14df3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T05:27:10.158 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:09 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:09.830+0000 7f8f14df3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T05:27:10.158 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:09 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:09.932+0000 7f8f14df3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T05:27:10.158 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:27:10 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:27:10.159+0000 7f8f14df3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T05:27:10.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:10 smithi023 ceph-mon[25869]: Active manager daemon y restarted 2024-03-24T05:27:10.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:10 smithi023 ceph-mon[25869]: Activating manager daemon y 2024-03-24T05:27:10.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:10 smithi023 ceph-mon[25869]: osdmap e3: 0 total, 0 up, 0 in 2024-03-24T05:27:10.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:10 smithi023 ceph-mon[25869]: mgrmap e10: y(active, starting, since 0.00791161s) 2024-03-24T05:27:10.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:10 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T05:27:10.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:10 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T05:27:10.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:10 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T05:27:10.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:10 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T05:27:10.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:10 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T05:27:10.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:10 smithi023 ceph-mon[25869]: Manager daemon y is now available 2024-03-24T05:27:11.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:11 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:27:11.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:11 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:27:11.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:11 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T05:27:11.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:11 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T05:27:11.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:11 smithi023 ceph-mon[25869]: [24/Mar/2024:05:27:10] ENGINE Bus STARTING 2024-03-24T05:27:11.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:11 smithi023 ceph-mon[25869]: [24/Mar/2024:05:27:10] ENGINE Serving on https://172.21.15.23:7150 2024-03-24T05:27:11.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:11 smithi023 ceph-mon[25869]: [24/Mar/2024:05:27:10] ENGINE Serving on http://172.21.15.23:8765 2024-03-24T05:27:11.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:11 smithi023 ceph-mon[25869]: [24/Mar/2024:05:27:10] ENGINE Bus STARTED 2024-03-24T05:27:11.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:11 smithi023 ceph-mon[25869]: mgrmap e11: y(active, since 1.00572s) 2024-03-24T05:27:11.753 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-24T05:27:11.753 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-24T05:27:11.753 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T05:27:11.753 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-24T05:27:11.753 INFO:teuthology.orchestra.run.smithi023.stdout:mgr epoch 9 is available 2024-03-24T05:27:11.753 INFO:teuthology.orchestra.run.smithi023.stdout:Generating a dashboard self-signed certificate... 2024-03-24T05:27:12.323 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:12 smithi023 ceph-mon[25869]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T05:27:12.323 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:12 smithi023 ceph-mon[25869]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T05:27:13.383 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:13 smithi023 ceph-mon[25869]: mgrmap e12: y(active, since 2s) 2024-03-24T05:27:13.383 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:13 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:13.383 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:13 smithi023 ceph-mon[25869]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:27:13.383 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:13 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:13.383 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:13 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:13.385 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-24T05:27:13.385 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial admin user... 2024-03-24T05:27:14.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:14 smithi023 ceph-mon[25869]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:27:14.841 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$L7dJHurnpR5pKaSiCNX9p.4YARB9g4i6ZJgmslebGl5O8lWZ9k/zi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711258034, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-24T05:27:14.841 INFO:teuthology.orchestra.run.smithi023.stdout:Fetching dashboard port number... 2024-03-24T05:27:15.514 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:15 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:15.515 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:15 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:16.431 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:16 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/4062488275' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-24T05:27:16.434 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 8443 2024-03-24T05:27:16.443 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T05:27:16.443 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-24T05:27:16.451 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T05:27:16.451 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-24T05:27:16.451 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-24T05:27:16.451 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-24T05:27:16.453 INFO:teuthology.orchestra.run.smithi023.stdout:Ceph Dashboard is now available at: 2024-03-24T05:27:16.453 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:27:16.453 INFO:teuthology.orchestra.run.smithi023.stdout: URL: https://smithi023.front.sepia.ceph.com:8443/ 2024-03-24T05:27:16.453 INFO:teuthology.orchestra.run.smithi023.stdout: User: admin 2024-03-24T05:27:16.453 INFO:teuthology.orchestra.run.smithi023.stdout: Password: kqzlqvzqyi 2024-03-24T05:27:16.454 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:27:16.454 INFO:teuthology.orchestra.run.smithi023.stdout:Saving cluster configuration to /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config directory 2024-03-24T05:27:16.454 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling autotune for osd_memory_target 2024-03-24T05:27:17.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:17 smithi023 ceph-mon[25869]: mgrmap e13: y(active, since 6s) 2024-03-24T05:27:19.581 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-24T05:27:19.582 INFO:teuthology.orchestra.run.smithi023.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-24T05:27:19.582 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:27:19.582 INFO:teuthology.orchestra.run.smithi023.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-24T05:27:19.582 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:27:19.582 INFO:teuthology.orchestra.run.smithi023.stdout:Or, if you are only running a single cluster on this host: 2024-03-24T05:27:19.582 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:27:19.582 INFO:teuthology.orchestra.run.smithi023.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-24T05:27:19.583 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:27:19.583 INFO:teuthology.orchestra.run.smithi023.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-24T05:27:19.583 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:27:19.583 INFO:teuthology.orchestra.run.smithi023.stdout: ceph telemetry on 2024-03-24T05:27:19.583 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:27:19.583 INFO:teuthology.orchestra.run.smithi023.stdout:For more information see: 2024-03-24T05:27:19.583 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:27:19.583 INFO:teuthology.orchestra.run.smithi023.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-24T05:27:19.583 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:27:19.583 INFO:teuthology.orchestra.run.smithi023.stdout:Bootstrap complete. 2024-03-24T05:27:19.628 INFO:tasks.cephadm:Fetching config... 2024-03-24T05:27:19.628 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T05:27:19.628 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-24T05:27:19.645 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-24T05:27:19.645 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T05:27:19.645 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-24T05:27:19.700 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-24T05:27:19.700 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T05:27:19.700 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-24T05:27:19.767 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-24T05:27:19.767 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T05:27:19.767 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-24T05:27:19.821 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-24T05:27:19.822 DEBUG:teuthology.orchestra.run.smithi023:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvJKTk74/y/amuZCo0pwTTDWxkyr9ZsDKXDk3UGFWuG0p4rvKVduZCjM3LknLU/IAxV/52nb0YnyImq+BrcrGgpOWh7hqDaISdjx2bMk6C3VSLgGTz4jYeB0uwiaCSUJ8+0uSshwWgW08E2N/4xX5x8o4q3zZwPF0rDRcCp59D/KSoTDZY/pxwfmGHy6p2EI2UwExUrdtYMzeQFEMJ8AyG5tv8weRhk17ZAIqUohKH6Z/8n0Mt/yppEUledBhFrayhEKVg1LPmihQn+uOrTdMcBxa0SAFI3MDdwuPALmPKCkM8XBW2qVqa/MgczaiGYwFVchd9uPjl9wUBKccUNnbo/LcrXEs4KxOOHqwFx4pu+XSKehow9Nrmey2G6LNgMrmnLF4fT5V55s+cRNOpJlxBdK2/BJRsmbE9TynlCuvQ0+jr7dNyHYKq+PtoGev5667m09bp+6Yrx1aKEck2jcyt2fWwybkBhmUndbWCkgllTdYt/vq83mwWx4f/1gDt6KE= ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T05:27:19.910 INFO:teuthology.orchestra.run.smithi023.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvJKTk74/y/amuZCo0pwTTDWxkyr9ZsDKXDk3UGFWuG0p4rvKVduZCjM3LknLU/IAxV/52nb0YnyImq+BrcrGgpOWh7hqDaISdjx2bMk6C3VSLgGTz4jYeB0uwiaCSUJ8+0uSshwWgW08E2N/4xX5x8o4q3zZwPF0rDRcCp59D/KSoTDZY/pxwfmGHy6p2EI2UwExUrdtYMzeQFEMJ8AyG5tv8weRhk17ZAIqUohKH6Z/8n0Mt/yppEUledBhFrayhEKVg1LPmihQn+uOrTdMcBxa0SAFI3MDdwuPALmPKCkM8XBW2qVqa/MgczaiGYwFVchd9uPjl9wUBKccUNnbo/LcrXEs4KxOOHqwFx4pu+XSKehow9Nrmey2G6LNgMrmnLF4fT5V55s+cRNOpJlxBdK2/BJRsmbE9TynlCuvQ0+jr7dNyHYKq+PtoGev5667m09bp+6Yrx1aKEck2jcyt2fWwybkBhmUndbWCkgllTdYt/vq83mwWx4f/1gDt6KE= ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:27:19.934 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvJKTk74/y/amuZCo0pwTTDWxkyr9ZsDKXDk3UGFWuG0p4rvKVduZCjM3LknLU/IAxV/52nb0YnyImq+BrcrGgpOWh7hqDaISdjx2bMk6C3VSLgGTz4jYeB0uwiaCSUJ8+0uSshwWgW08E2N/4xX5x8o4q3zZwPF0rDRcCp59D/KSoTDZY/pxwfmGHy6p2EI2UwExUrdtYMzeQFEMJ8AyG5tv8weRhk17ZAIqUohKH6Z/8n0Mt/yppEUledBhFrayhEKVg1LPmihQn+uOrTdMcBxa0SAFI3MDdwuPALmPKCkM8XBW2qVqa/MgczaiGYwFVchd9uPjl9wUBKccUNnbo/LcrXEs4KxOOHqwFx4pu+XSKehow9Nrmey2G6LNgMrmnLF4fT5V55s+cRNOpJlxBdK2/BJRsmbE9TynlCuvQ0+jr7dNyHYKq+PtoGev5667m09bp+6Yrx1aKEck2jcyt2fWwybkBhmUndbWCkgllTdYt/vq83mwWx4f/1gDt6KE= ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T05:27:19.980 INFO:teuthology.orchestra.run.smithi063.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvJKTk74/y/amuZCo0pwTTDWxkyr9ZsDKXDk3UGFWuG0p4rvKVduZCjM3LknLU/IAxV/52nb0YnyImq+BrcrGgpOWh7hqDaISdjx2bMk6C3VSLgGTz4jYeB0uwiaCSUJ8+0uSshwWgW08E2N/4xX5x8o4q3zZwPF0rDRcCp59D/KSoTDZY/pxwfmGHy6p2EI2UwExUrdtYMzeQFEMJ8AyG5tv8weRhk17ZAIqUohKH6Z/8n0Mt/yppEUledBhFrayhEKVg1LPmihQn+uOrTdMcBxa0SAFI3MDdwuPALmPKCkM8XBW2qVqa/MgczaiGYwFVchd9uPjl9wUBKccUNnbo/LcrXEs4KxOOHqwFx4pu+XSKehow9Nrmey2G6LNgMrmnLF4fT5V55s+cRNOpJlxBdK2/BJRsmbE9TynlCuvQ0+jr7dNyHYKq+PtoGev5667m09bp+6Yrx1aKEck2jcyt2fWwybkBhmUndbWCkgllTdYt/vq83mwWx4f/1gDt6KE= ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:27:19.995 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvJKTk74/y/amuZCo0pwTTDWxkyr9ZsDKXDk3UGFWuG0p4rvKVduZCjM3LknLU/IAxV/52nb0YnyImq+BrcrGgpOWh7hqDaISdjx2bMk6C3VSLgGTz4jYeB0uwiaCSUJ8+0uSshwWgW08E2N/4xX5x8o4q3zZwPF0rDRcCp59D/KSoTDZY/pxwfmGHy6p2EI2UwExUrdtYMzeQFEMJ8AyG5tv8weRhk17ZAIqUohKH6Z/8n0Mt/yppEUledBhFrayhEKVg1LPmihQn+uOrTdMcBxa0SAFI3MDdwuPALmPKCkM8XBW2qVqa/MgczaiGYwFVchd9uPjl9wUBKccUNnbo/LcrXEs4KxOOHqwFx4pu+XSKehow9Nrmey2G6LNgMrmnLF4fT5V55s+cRNOpJlxBdK2/BJRsmbE9TynlCuvQ0+jr7dNyHYKq+PtoGev5667m09bp+6Yrx1aKEck2jcyt2fWwybkBhmUndbWCkgllTdYt/vq83mwWx4f/1gDt6KE= ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T05:27:20.042 INFO:teuthology.orchestra.run.smithi143.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvJKTk74/y/amuZCo0pwTTDWxkyr9ZsDKXDk3UGFWuG0p4rvKVduZCjM3LknLU/IAxV/52nb0YnyImq+BrcrGgpOWh7hqDaISdjx2bMk6C3VSLgGTz4jYeB0uwiaCSUJ8+0uSshwWgW08E2N/4xX5x8o4q3zZwPF0rDRcCp59D/KSoTDZY/pxwfmGHy6p2EI2UwExUrdtYMzeQFEMJ8AyG5tv8weRhk17ZAIqUohKH6Z/8n0Mt/yppEUledBhFrayhEKVg1LPmihQn+uOrTdMcBxa0SAFI3MDdwuPALmPKCkM8XBW2qVqa/MgczaiGYwFVchd9uPjl9wUBKccUNnbo/LcrXEs4KxOOHqwFx4pu+XSKehow9Nrmey2G6LNgMrmnLF4fT5V55s+cRNOpJlxBdK2/BJRsmbE9TynlCuvQ0+jr7dNyHYKq+PtoGev5667m09bp+6Yrx1aKEck2jcyt2fWwybkBhmUndbWCkgllTdYt/vq83mwWx4f/1gDt6KE= ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:27:20.057 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-24T05:27:20.139 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:19 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/789177062' entity='client.admin' 2024-03-24T05:27:20.300 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:27:23.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:22 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:23.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:22 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:23.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:22 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi023", "name": "osd_memory_target"}]: dispatch 2024-03-24T05:27:23.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:22 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:23.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:22 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:27:23.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:22 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:23.383 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-24T05:27:23.383 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-24T05:27:23.621 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:27:24.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:23 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/2510021660' entity='client.admin' 2024-03-24T05:27:24.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:23 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:27:24.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:23 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:25.962 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi063 2024-03-24T05:27:25.963 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-24T05:27:25.963 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/etc/ceph/ceph.conf 2024-03-24T05:27:25.982 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-24T05:27:25.982 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T05:27:26.039 INFO:tasks.cephadm:Adding host smithi063 to orchestrator... 2024-03-24T05:27:26.040 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph orch host add smithi063 2024-03-24T05:27:26.277 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:27:26.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:26 smithi023 ceph-mon[25869]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:27:26.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:26 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:26.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:26 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:27:26.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:26 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:27:26.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:26 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:27:26.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:26 smithi023 ceph-mon[25869]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-24T05:27:26.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:26 smithi023 ceph-mon[25869]: Updating smithi023:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:27:26.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:26 smithi023 ceph-mon[25869]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-24T05:27:26.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:26 smithi023 ceph-mon[25869]: Updating smithi023:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T05:27:26.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:26 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:26.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:26 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:26.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:26 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:29.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:29 smithi023 ceph-mon[25869]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi063", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:27:29.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:29 smithi023 ceph-mon[25869]: Deploying cephadm binary to smithi063 2024-03-24T05:27:31.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:31 smithi023 ceph-mon[25869]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:32.546 INFO:teuthology.orchestra.run.smithi023.stdout:Added host 'smithi063' with addr '172.21.15.63' 2024-03-24T05:27:33.147 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T05:27:33.386 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:27:33.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:33 smithi023 ceph-mon[25869]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:33.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:33 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:33.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:33 smithi023 ceph-mon[25869]: Added host smithi063 2024-03-24T05:27:33.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:33 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:27:33.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:33 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:34.644 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:34 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:35.084 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:27:35.085 INFO:teuthology.orchestra.run.smithi023.stdout:[{"addr": "172.21.15.23", "hostname": "smithi023", "labels": [], "status": ""}, {"addr": "172.21.15.63", "hostname": "smithi063", "labels": [], "status": ""}] 2024-03-24T05:27:35.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:35 smithi023 ceph-mon[25869]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:35.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:35 smithi023 ceph-mon[25869]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T05:27:35.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:35 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:35.814 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi143 2024-03-24T05:27:35.814 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T05:27:35.814 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/etc/ceph/ceph.conf 2024-03-24T05:27:35.833 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T05:27:35.833 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T05:27:35.891 INFO:tasks.cephadm:Adding host smithi143 to orchestrator... 2024-03-24T05:27:35.891 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph orch host add smithi143 2024-03-24T05:27:36.134 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:27:37.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:37 smithi023 ceph-mon[25869]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:39.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:38 smithi023 ceph-mon[25869]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi143", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:27:40.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:39 smithi023 ceph-mon[25869]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:40.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:39 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:40.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:39 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:40.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:39 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:40.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:39 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:40.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:39 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi063", "name": "osd_memory_target"}]: dispatch 2024-03-24T05:27:40.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:39 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:27:40.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:39 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:27:40.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:39 smithi023 ceph-mon[25869]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-24T05:27:40.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:39 smithi023 ceph-mon[25869]: Deploying cephadm binary to smithi143 2024-03-24T05:27:40.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:39 smithi023 ceph-mon[25869]: Updating smithi063:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:27:40.083 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:39 smithi023 ceph-mon[25869]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-03-24T05:27:41.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:40 smithi023 ceph-mon[25869]: Updating smithi063:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T05:27:42.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:41 smithi023 ceph-mon[25869]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:42.433 INFO:teuthology.orchestra.run.smithi023.stdout:Added host 'smithi143' with addr '172.21.15.143' 2024-03-24T05:27:43.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:42 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:43.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:42 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:43.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:42 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:43.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:42 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:43.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:42 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:27:43.232 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T05:27:43.468 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:27:44.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:43 smithi023 ceph-mon[25869]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:44.270 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:43 smithi023 ceph-mon[25869]: Added host smithi143 2024-03-24T05:27:44.270 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:43 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:44.270 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:43 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:45.195 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:27:45.196 INFO:teuthology.orchestra.run.smithi023.stdout:[{"addr": "172.21.15.23", "hostname": "smithi023", "labels": [], "status": ""}, {"addr": "172.21.15.63", "hostname": "smithi063", "labels": [], "status": ""}, {"addr": "172.21.15.143", "hostname": "smithi143", "labels": [], "status": ""}] 2024-03-24T05:27:45.677 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-24T05:27:45.677 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-24T05:27:45.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:45 smithi023 ceph-mon[25869]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:45.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:45 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:45.912 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:27:46.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:46 smithi023 ceph-mon[25869]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T05:27:47.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:47 smithi023 ceph-mon[25869]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:48.529 INFO:teuthology.orchestra.run.smithi023.stderr:adjusted tunables profile to default 2024-03-24T05:27:48.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:48 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/2816691626' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T05:27:48.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:48 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:48.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:48 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:48.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:48 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:48.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:48 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:48.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:48 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi143", "name": "osd_memory_target"}]: dispatch 2024-03-24T05:27:48.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:48 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:27:48.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:48 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:27:48.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:48 smithi023 ceph-mon[25869]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T05:27:48.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:48 smithi023 ceph-mon[25869]: Updating smithi143:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:27:49.129 INFO:tasks.cephadm:Adding mon.a on smithi023 2024-03-24T05:27:49.130 INFO:tasks.cephadm:Adding mon.b on smithi063 2024-03-24T05:27:49.130 INFO:tasks.cephadm:Adding mon.c on smithi143 2024-03-24T05:27:49.130 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi023:172.21.15.23=a;smithi063:172.21.15.63=b;smithi143:172.21.15.143=c' 2024-03-24T05:27:49.355 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:27:49.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:49 smithi023 ceph-mon[25869]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:49.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:49 smithi023 ceph-mon[25869]: Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-03-24T05:27:49.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:49 smithi023 ceph-mon[25869]: Updating smithi143:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T05:27:49.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:49 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/2816691626' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T05:27:49.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:49 smithi023 ceph-mon[25869]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T05:27:49.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:49 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:49.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:49 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:49.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:49 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:50.991 INFO:teuthology.orchestra.run.smithi143.stdout:Scheduled mon update... 2024-03-24T05:27:51.610 DEBUG:teuthology.orchestra.run.smithi063:mon.b> sudo journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mon.b.service 2024-03-24T05:27:51.612 DEBUG:teuthology.orchestra.run.smithi143:mon.c> sudo journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mon.c.service 2024-03-24T05:27:51.614 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T05:27:51.615 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T05:27:52.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:51 smithi023 ceph-mon[25869]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:52.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:51 smithi023 ceph-mon[25869]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi063:172.21.15.63=b;smithi143:172.21.15.143=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:27:52.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:51 smithi023 ceph-mon[25869]: Saving service mon spec with placement smithi023:172.21.15.23=a;smithi063:172.21.15.63=b;smithi143:172.21.15.143=c;count:3 2024-03-24T05:27:52.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:51 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:52.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:51 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:27:52.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:51 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:27:52.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:51 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:27:52.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:51 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:52.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:51 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T05:27:52.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:51 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:27:52.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:51 smithi023 ceph-mon[25869]: Deploying daemon mon.c on smithi143 2024-03-24T05:27:52.608 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:27:54.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:53 smithi023 ceph-mon[25869]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:55.612 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T05:27:55.612 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":1,"fsid":"e2aa55c2-e99e-11ee-95cd-87774f69a715","modified":"2024-03-24T05:26:07.735023Z","created":"2024-03-24T05:26:07.735023Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T05:27:55.613 INFO:teuthology.orchestra.run.smithi143.stderr:dumped monmap epoch 1 2024-03-24T05:27:55.924 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:55 smithi143 systemd[1]: Starting Ceph mon.c for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:27:56.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:55 smithi023 ceph-mon[25869]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:56.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:27:55 smithi023 ceph-mon[25869]: from='client.? 172.21.15.143:0/2441656539' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T05:27:56.486 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 podman[30250]: 2024-03-24 05:27:56.31857231 +0000 UTC m=+0.092502145 container create 28c1a290e01f7ae2a92ff3b85ef78f1b963c2f8c5e15383dde992b53fb66380a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-c, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-24T05:27:56.486 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 podman[30250]: 2024-03-24 05:27:56.243370004 +0000 UTC m=+0.017299808 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T05:27:56.487 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 podman[30250]: 2024-03-24 05:27:56.48739766 +0000 UTC m=+0.261327464 container init 28c1a290e01f7ae2a92ff3b85ef78f1b963c2f8c5e15383dde992b53fb66380a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-24T05:27:56.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 podman[30250]: 2024-03-24 05:27:56.493882079 +0000 UTC m=+0.267811868 container start 28c1a290e01f7ae2a92ff3b85ef78f1b963c2f8c5e15383dde992b53fb66380a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-24T05:27:56.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T05:27:56.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 7 2024-03-24T05:27:56.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: pidfile_write: ignore empty --pid-file 2024-03-24T05:27:56.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: load: jerasure load: lrc 2024-03-24T05:27:56.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: RocksDB version: 7.9.2 2024-03-24T05:27:56.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Git sha 0 2024-03-24T05:27:56.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T05:27:56.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: DB SUMMARY 2024-03-24T05:27:56.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: DB Session ID: XESLH10AJRAOFGY8UMMM 2024-03-24T05:27:56.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: CURRENT file: CURRENT 2024-03-24T05:27:56.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T05:27:56.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-24T05:27:56.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-24T05:27:56.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-24T05:27:56.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.error_if_exists: 0 2024-03-24T05:27:56.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.create_if_missing: 0 2024-03-24T05:27:56.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.paranoid_checks: 1 2024-03-24T05:27:56.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T05:27:56.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T05:27:56.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T05:27:56.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.env: 0x55ccfb286ae0 2024-03-24T05:27:56.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T05:27:56.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.info_log: 0x55ccfc05d8c0 2024-03-24T05:27:56.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T05:27:56.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.statistics: (nil) 2024-03-24T05:27:56.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.use_fsync: 0 2024-03-24T05:27:56.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_log_file_size: 0 2024-03-24T05:27:56.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T05:27:56.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T05:27:56.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T05:27:56.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T05:27:56.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.allow_fallocate: 1 2024-03-24T05:27:56.813 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T05:27:56.813 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T05:27:56.814 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.use_direct_reads: 0 2024-03-24T05:27:56.814 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T05:27:56.814 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T05:27:56.814 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.db_log_dir: 2024-03-24T05:27:56.814 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.wal_dir: 2024-03-24T05:27:56.814 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T05:27:56.814 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T05:27:56.814 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T05:27:56.815 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T05:27:56.815 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T05:27:56.815 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T05:27:56.815 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T05:27:56.815 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T05:27:56.815 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.write_buffer_manager: 0x55ccfc031400 2024-03-24T05:27:56.815 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T05:27:56.815 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T05:27:56.815 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T05:27:56.815 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T05:27:56.816 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T05:27:56.816 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T05:27:56.816 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T05:27:56.816 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T05:27:56.816 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.unordered_write: 0 2024-03-24T05:27:56.816 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T05:27:56.816 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T05:27:56.816 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T05:27:56.816 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T05:27:56.816 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.row_cache: None 2024-03-24T05:27:56.817 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.wal_filter: None 2024-03-24T05:27:56.817 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T05:27:56.817 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T05:27:56.817 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.two_write_queues: 0 2024-03-24T05:27:56.817 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T05:27:56.817 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.wal_compression: 0 2024-03-24T05:27:56.817 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.atomic_flush: 0 2024-03-24T05:27:56.817 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T05:27:56.817 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T05:27:56.817 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T05:27:56.818 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.log_readahead_size: 0 2024-03-24T05:27:56.818 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T05:27:56.818 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T05:27:56.818 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T05:27:56.818 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T05:27:56.818 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T05:27:56.818 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T05:27:56.818 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T05:27:56.819 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_background_jobs: 2 2024-03-24T05:27:56.819 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_background_compactions: -1 2024-03-24T05:27:56.819 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_subcompactions: 1 2024-03-24T05:27:56.819 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T05:27:56.819 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T05:27:56.819 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T05:27:56.819 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T05:27:56.819 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T05:27:56.819 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T05:27:56.819 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T05:27:56.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T05:27:56.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_open_files: -1 2024-03-24T05:27:56.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T05:27:56.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T05:27:56.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T05:27:56.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T05:27:56.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_background_flushes: -1 2024-03-24T05:27:56.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Compression algorithms supported: 2024-03-24T05:27:56.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T05:27:56.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: kZSTD supported: 0 2024-03-24T05:27:56.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: kXpressCompression supported: 0 2024-03-24T05:27:56.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T05:27:56.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: kZlibCompression supported: 1 2024-03-24T05:27:56.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: kSnappyCompression supported: 1 2024-03-24T05:27:56.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: kLZ4Compression supported: 1 2024-03-24T05:27:56.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: kBZip2Compression supported: 0 2024-03-24T05:27:56.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T05:27:56.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T05:27:56.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-24T05:27:56.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T05:27:56.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T05:27:56.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.merge_operator: 2024-03-24T05:27:56.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compaction_filter: None 2024-03-24T05:27:56.822 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T05:27:56.822 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T05:27:56.822 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T05:27:56.822 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T05:27:56.822 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ccfc05d9c0) 2024-03-24T05:27:56.822 INFO:journalctl@ceph.mon.c.smithi143.stdout: cache_index_and_filter_blocks: 1 2024-03-24T05:27:56.822 INFO:journalctl@ceph.mon.c.smithi143.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T05:27:56.822 INFO:journalctl@ceph.mon.c.smithi143.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T05:27:56.822 INFO:journalctl@ceph.mon.c.smithi143.stdout: pin_top_level_index_and_filter: 1 2024-03-24T05:27:56.822 INFO:journalctl@ceph.mon.c.smithi143.stdout: index_type: 0 2024-03-24T05:27:56.823 INFO:journalctl@ceph.mon.c.smithi143.stdout: data_block_index_type: 0 2024-03-24T05:27:56.823 INFO:journalctl@ceph.mon.c.smithi143.stdout: index_shortening: 1 2024-03-24T05:27:56.823 INFO:journalctl@ceph.mon.c.smithi143.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T05:27:56.823 INFO:journalctl@ceph.mon.c.smithi143.stdout: checksum: 4 2024-03-24T05:27:56.823 INFO:journalctl@ceph.mon.c.smithi143.stdout: no_block_cache: 0 2024-03-24T05:27:56.823 INFO:journalctl@ceph.mon.c.smithi143.stdout: block_cache: 0x55ccfc061090 2024-03-24T05:27:56.823 INFO:journalctl@ceph.mon.c.smithi143.stdout: block_cache_name: BinnedLRUCache 2024-03-24T05:27:56.823 INFO:journalctl@ceph.mon.c.smithi143.stdout: block_cache_options: 2024-03-24T05:27:56.823 INFO:journalctl@ceph.mon.c.smithi143.stdout: capacity : 536870912 2024-03-24T05:27:56.823 INFO:journalctl@ceph.mon.c.smithi143.stdout: num_shard_bits : 4 2024-03-24T05:27:56.824 INFO:journalctl@ceph.mon.c.smithi143.stdout: strict_capacity_limit : 0 2024-03-24T05:27:56.824 INFO:journalctl@ceph.mon.c.smithi143.stdout: high_pri_pool_ratio: 0.000 2024-03-24T05:27:56.824 INFO:journalctl@ceph.mon.c.smithi143.stdout: block_cache_compressed: (nil) 2024-03-24T05:27:56.824 INFO:journalctl@ceph.mon.c.smithi143.stdout: persistent_cache: (nil) 2024-03-24T05:27:56.824 INFO:journalctl@ceph.mon.c.smithi143.stdout: block_size: 4096 2024-03-24T05:27:56.824 INFO:journalctl@ceph.mon.c.smithi143.stdout: block_size_deviation: 10 2024-03-24T05:27:56.824 INFO:journalctl@ceph.mon.c.smithi143.stdout: block_restart_interval: 16 2024-03-24T05:27:56.824 INFO:journalctl@ceph.mon.c.smithi143.stdout: index_block_restart_interval: 1 2024-03-24T05:27:56.824 INFO:journalctl@ceph.mon.c.smithi143.stdout: metadata_block_size: 4096 2024-03-24T05:27:56.824 INFO:journalctl@ceph.mon.c.smithi143.stdout: partition_filters: 0 2024-03-24T05:27:56.824 INFO:journalctl@ceph.mon.c.smithi143.stdout: use_delta_encoding: 1 2024-03-24T05:27:56.825 INFO:journalctl@ceph.mon.c.smithi143.stdout: filter_policy: bloomfilter 2024-03-24T05:27:56.825 INFO:journalctl@ceph.mon.c.smithi143.stdout: whole_key_filtering: 1 2024-03-24T05:27:56.825 INFO:journalctl@ceph.mon.c.smithi143.stdout: verify_compression: 0 2024-03-24T05:27:56.825 INFO:journalctl@ceph.mon.c.smithi143.stdout: read_amp_bytes_per_bit: 0 2024-03-24T05:27:56.825 INFO:journalctl@ceph.mon.c.smithi143.stdout: format_version: 5 2024-03-24T05:27:56.825 INFO:journalctl@ceph.mon.c.smithi143.stdout: enable_index_compression: 1 2024-03-24T05:27:56.825 INFO:journalctl@ceph.mon.c.smithi143.stdout: block_align: 0 2024-03-24T05:27:56.825 INFO:journalctl@ceph.mon.c.smithi143.stdout: max_auto_readahead_size: 262144 2024-03-24T05:27:56.825 INFO:journalctl@ceph.mon.c.smithi143.stdout: prepopulate_block_cache: 0 2024-03-24T05:27:56.825 INFO:journalctl@ceph.mon.c.smithi143.stdout: initial_auto_readahead_size: 8192 2024-03-24T05:27:56.825 INFO:journalctl@ceph.mon.c.smithi143.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T05:27:56.825 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T05:27:56.826 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T05:27:56.826 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compression: NoCompression 2024-03-24T05:27:56.826 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T05:27:56.826 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T05:27:56.826 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T05:27:56.826 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.num_levels: 7 2024-03-24T05:27:56.826 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T05:27:56.826 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T05:27:56.826 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T05:27:56.827 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T05:27:56.827 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T05:27:56.827 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T05:27:56.827 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T05:27:56.827 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T05:27:56.827 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T05:27:56.827 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T05:27:56.827 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T05:27:56.827 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T05:27:56.827 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T05:27:56.827 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T05:27:56.828 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T05:27:56.828 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T05:27:56.828 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T05:27:56.828 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T05:27:56.828 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T05:27:56.828 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T05:27:56.828 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T05:27:56.828 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T05:27:56.828 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T05:27:56.828 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T05:27:56.828 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T05:27:56.829 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T05:27:56.829 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T05:27:56.829 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T05:27:56.829 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T05:27:56.829 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T05:27:56.829 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T05:27:56.829 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T05:27:56.829 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T05:27:56.829 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T05:27:56.829 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T05:27:56.829 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T05:27:56.829 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T05:27:56.830 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T05:27:56.830 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T05:27:56.830 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T05:27:56.830 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T05:27:56.830 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T05:27:56.830 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T05:27:56.830 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T05:27:56.830 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T05:27:56.830 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T05:27:56.830 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T05:27:56.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T05:27:56.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T05:27:56.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T05:27:56.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T05:27:56.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T05:27:56.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T05:27:56.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.table_properties_collectors: 2024-03-24T05:27:56.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.inplace_update_support: 0 2024-03-24T05:27:56.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T05:27:56.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T05:27:56.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T05:27:56.832 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T05:27:56.832 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.bloom_locality: 0 2024-03-24T05:27:56.832 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.max_successive_merges: 0 2024-03-24T05:27:56.832 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T05:27:56.832 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T05:27:56.832 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T05:27:56.832 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T05:27:56.832 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.ttl: 2592000 2024-03-24T05:27:56.832 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T05:27:56.832 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T05:27:56.832 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T05:27:56.832 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.enable_blob_files: false 2024-03-24T05:27:56.833 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.min_blob_size: 0 2024-03-24T05:27:56.833 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T05:27:56.833 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T05:27:56.833 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T05:27:56.833 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T05:27:56.833 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T05:27:56.833 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T05:27:56.833 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T05:27:56.833 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T05:27:56.833 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T05:27:56.833 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-24T05:27:56.834 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 39811fd3-fbcf-435a-8f8c-cd8e7d883b21 2024-03-24T05:27:56.834 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711258076523488, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-24T05:27:56.834 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-24T05:27:56.834 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711258076523912, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711258076, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "39811fd3-fbcf-435a-8f8c-cd8e7d883b21", "db_session_id": "XESLH10AJRAOFGY8UMMM", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-24T05:27:56.834 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711258076523993, "job": 1, "event": "recovery_finished"} 2024-03-24T05:27:56.834 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-24T05:27:56.834 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T05:27:56.834 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ccfc152000 2024-03-24T05:27:56.834 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: DB pointer 0x55ccfc13c000 2024-03-24T05:27:56.834 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T05:27:56.834 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T05:27:56.835 INFO:journalctl@ceph.mon.c.smithi143.stdout: ** DB Stats ** 2024-03-24T05:27:56.835 INFO:journalctl@ceph.mon.c.smithi143.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T05:27:56.835 INFO:journalctl@ceph.mon.c.smithi143.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T05:27:56.835 INFO:journalctl@ceph.mon.c.smithi143.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T05:27:56.835 INFO:journalctl@ceph.mon.c.smithi143.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T05:27:56.835 INFO:journalctl@ceph.mon.c.smithi143.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T05:27:56.835 INFO:journalctl@ceph.mon.c.smithi143.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T05:27:56.835 INFO:journalctl@ceph.mon.c.smithi143.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T05:27:56.835 INFO:journalctl@ceph.mon.c.smithi143.stdout: 2024-03-24T05:27:56.835 INFO:journalctl@ceph.mon.c.smithi143.stdout: ** Compaction Stats [default] ** 2024-03-24T05:27:56.835 INFO:journalctl@ceph.mon.c.smithi143.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T05:27:56.836 INFO:journalctl@ceph.mon.c.smithi143.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T05:27:56.836 INFO:journalctl@ceph.mon.c.smithi143.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T05:27:56.836 INFO:journalctl@ceph.mon.c.smithi143.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T05:27:56.836 INFO:journalctl@ceph.mon.c.smithi143.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T05:27:56.836 INFO:journalctl@ceph.mon.c.smithi143.stdout: 2024-03-24T05:27:56.836 INFO:journalctl@ceph.mon.c.smithi143.stdout: ** Compaction Stats [default] ** 2024-03-24T05:27:56.836 INFO:journalctl@ceph.mon.c.smithi143.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T05:27:56.836 INFO:journalctl@ceph.mon.c.smithi143.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T05:27:56.836 INFO:journalctl@ceph.mon.c.smithi143.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T05:27:56.836 INFO:journalctl@ceph.mon.c.smithi143.stdout: 2024-03-24T05:27:56.836 INFO:journalctl@ceph.mon.c.smithi143.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T05:27:56.836 INFO:journalctl@ceph.mon.c.smithi143.stdout: 2024-03-24T05:27:56.837 INFO:journalctl@ceph.mon.c.smithi143.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T05:27:56.837 INFO:journalctl@ceph.mon.c.smithi143.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T05:27:56.837 INFO:journalctl@ceph.mon.c.smithi143.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T05:27:56.837 INFO:journalctl@ceph.mon.c.smithi143.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T05:27:56.837 INFO:journalctl@ceph.mon.c.smithi143.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T05:27:56.837 INFO:journalctl@ceph.mon.c.smithi143.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T05:27:56.837 INFO:journalctl@ceph.mon.c.smithi143.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T05:27:56.837 INFO:journalctl@ceph.mon.c.smithi143.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T05:27:56.837 INFO:journalctl@ceph.mon.c.smithi143.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T05:27:56.837 INFO:journalctl@ceph.mon.c.smithi143.stdout: Block cache BinnedLRUCache@0x55ccfc061090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-24T05:27:56.837 INFO:journalctl@ceph.mon.c.smithi143.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-24T05:27:56.838 INFO:journalctl@ceph.mon.c.smithi143.stdout: 2024-03-24T05:27:56.838 INFO:journalctl@ceph.mon.c.smithi143.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T05:27:56.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-24T05:27:56.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: using public_addr v2:172.21.15.143:0/0 -> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-03-24T05:27:56.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: starting mon.c rank -1 at public addrs [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] at bind addrs [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:27:56.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: mon.c@-1(???) e0 preinit fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:27:56.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: mon.c@-1(synchronizing).mds e1 new map 2024-03-24T05:27:56.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-24T05:27:56.838 INFO:journalctl@ceph.mon.c.smithi143.stdout: e1 2024-03-24T05:27:56.838 INFO:journalctl@ceph.mon.c.smithi143.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T05:27:56.838 INFO:journalctl@ceph.mon.c.smithi143.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T05:27:56.839 INFO:journalctl@ceph.mon.c.smithi143.stdout: legacy client fscid: -1 2024-03-24T05:27:56.839 INFO:journalctl@ceph.mon.c.smithi143.stdout: 2024-03-24T05:27:56.839 INFO:journalctl@ceph.mon.c.smithi143.stdout: No filesystems configured 2024-03-24T05:27:56.839 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T05:27:56.839 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T05:27:56.839 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T05:27:56.839 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T05:27:56.839 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T05:27:56.839 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T05:27:56.839 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T05:27:56.839 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T05:27:56.839 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T05:27:56.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T05:27:56.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:56.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T05:27:56.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:56.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:56.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi143", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:27:56.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:56.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:56.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:56.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:56.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:56.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi063", "name": "osd_memory_target"}]: dispatch 2024-03-24T05:27:56.841 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:27:56.841 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:27:56.841 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-24T05:27:56.841 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: Deploying cephadm binary to smithi143 2024-03-24T05:27:56.841 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: Updating smithi063:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:27:56.841 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-03-24T05:27:56.841 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: Updating smithi063:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T05:27:56.841 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:56.841 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:56.841 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:56.841 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:56.841 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:56.842 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:27:56.842 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:56.842 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: Added host smithi143 2024-03-24T05:27:56.842 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:56.842 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:56.842 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:56.842 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:56.842 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T05:27:56.842 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:56.842 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/2816691626' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T05:27:56.842 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:56.842 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:56.843 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:56.843 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:56.843 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi143", "name": "osd_memory_target"}]: dispatch 2024-03-24T05:27:56.843 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:27:56.843 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:27:56.843 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T05:27:56.843 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: Updating smithi143:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:27:56.843 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:56.843 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-03-24T05:27:56.843 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: Updating smithi143:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T05:27:56.843 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/2816691626' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T05:27:56.843 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T05:27:56.844 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:56.844 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:56.844 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:56.844 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:56.844 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi063:172.21.15.63=b;smithi143:172.21.15.143=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:27:56.844 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: Saving service mon spec with placement smithi023:172.21.15.23=a;smithi063:172.21.15.63=b;smithi143:172.21.15.143=c;count:3 2024-03-24T05:27:56.844 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:56.844 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:27:56.844 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:27:56.844 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:27:56.844 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:27:56.844 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T05:27:56.845 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:27:56.845 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: Deploying daemon mon.c on smithi143 2024-03-24T05:27:56.845 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:56.845 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:27:56.845 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: from='client.? 172.21.15.143:0/2441656539' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T05:27:56.845 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 ceph-mon[30288]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-24T05:27:56.845 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 bash[30250]: 28c1a290e01f7ae2a92ff3b85ef78f1b963c2f8c5e15383dde992b53fb66380a 2024-03-24T05:27:56.845 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:27:56 smithi143 systemd[1]: Started Ceph mon.c for e2aa55c2-e99e-11ee-95cd-87774f69a715. 2024-03-24T05:27:57.179 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T05:27:57.179 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T05:27:57.413 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T05:28:01.988 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:01.988 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T05:28:01.988 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: mon.a calling monitor election 2024-03-24T05:28:01.988 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:28:01.988 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:28:01.988 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:01.988 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:28:01.989 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: mon.c calling monitor election 2024-03-24T05:28:01.989 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:28:01.989 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:01.989 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:28:01.989 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:28:01.989 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T05:28:01.989 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: monmap e2: 2 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],c=[v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T05:28:01.989 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: fsmap 2024-03-24T05:28:01.989 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T05:28:01.990 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: mgrmap e13: y(active, since 51s) 2024-03-24T05:28:01.990 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: overall HEALTH_OK 2024-03-24T05:28:01.990 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:01.990 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:01.990 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:01.990 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T05:28:01.990 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:01 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:02.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:02.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T05:28:02.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: mon.a calling monitor election 2024-03-24T05:28:02.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:28:02.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:28:02.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:02.059 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:28:02.059 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: mon.c calling monitor election 2024-03-24T05:28:02.059 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:28:02.059 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:02.059 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:28:02.059 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:28:02.059 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T05:28:02.059 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: monmap e2: 2 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],c=[v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T05:28:02.059 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: fsmap 2024-03-24T05:28:02.059 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T05:28:02.060 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: mgrmap e13: y(active, since 51s) 2024-03-24T05:28:02.060 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: overall HEALTH_OK 2024-03-24T05:28:02.060 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:02.060 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:02.060 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:02.060 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T05:28:02.060 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:01 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:02.262 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T05:28:02.263 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":2,"fsid":"e2aa55c2-e99e-11ee-95cd-87774f69a715","modified":"2024-03-24T05:27:56.547612Z","created":"2024-03-24T05:26:07.735023Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T05:28:02.264 INFO:teuthology.orchestra.run.smithi143.stderr:dumped monmap epoch 2 2024-03-24T05:28:02.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:02 smithi143 ceph-mon[30288]: Deploying daemon mon.b on smithi063 2024-03-24T05:28:02.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:02 smithi143 ceph-mon[30288]: from='client.? 172.21.15.143:0/1016230467' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T05:28:02.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:02 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:28:02.910 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:02 smithi023 ceph-mon[25869]: Deploying daemon mon.b on smithi063 2024-03-24T05:28:02.910 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:02 smithi023 ceph-mon[25869]: from='client.? 172.21.15.143:0/1016230467' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T05:28:02.910 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:02 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:28:03.866 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T05:28:03.867 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T05:28:04.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:03 smithi143 ceph-mon[30288]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:04.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:03 smithi023 ceph-mon[25869]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:04.110 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T05:28:05.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:05 smithi063 ceph-mon[30595]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-24T05:28:06.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:05 smithi143 ceph-mon[30288]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:06.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:05 smithi023 ceph-mon[25869]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:06.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:05 smithi063 bash[30557]: 055378e48274efe9de3dcb07b05f5d9cd3bbf6e37676a4a64815252af62365f1 2024-03-24T05:28:06.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:05 smithi063 systemd[1]: Started Ceph mon.b for e2aa55c2-e99e-11ee-95cd-87774f69a715. 2024-03-24T05:28:10.709 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T05:28:10.710 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":3,"fsid":"e2aa55c2-e99e-11ee-95cd-87774f69a715","modified":"2024-03-24T05:28:05.675474Z","created":"2024-03-24T05:26:07.735023Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T05:28:10.713 INFO:teuthology.orchestra.run.smithi143.stderr:dumped monmap epoch 3 2024-03-24T05:28:11.034 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: mon.c calling monitor election 2024-03-24T05:28:11.034 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T05:28:11.034 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: mon.a calling monitor election 2024-03-24T05:28:11.034 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:11.034 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:28:11.034 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: from='client.? 172.21.15.143:0/3439166125' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T05:28:11.035 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:11.035 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:11.035 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:11.035 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:11.035 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:11.035 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:11.035 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:11.035 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T05:28:11.035 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0],c=[v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T05:28:11.035 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: fsmap 2024-03-24T05:28:11.035 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T05:28:11.036 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: mgrmap e13: y(active, since 60s) 2024-03-24T05:28:11.036 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-24T05:28:11.036 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T05:28:11.036 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T05:28:11.036 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: mon.b (rank 2) addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] is down (out of quorum) 2024-03-24T05:28:11.036 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:11.036 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:11.036 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:11.036 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:10 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:11.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: mon.c calling monitor election 2024-03-24T05:28:11.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T05:28:11.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: mon.a calling monitor election 2024-03-24T05:28:11.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:11.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:28:11.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: from='client.? 172.21.15.143:0/3439166125' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T05:28:11.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:11.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:11.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:11.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:11.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:11.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:11.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:11.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T05:28:11.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0],c=[v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T05:28:11.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: fsmap 2024-03-24T05:28:11.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T05:28:11.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: mgrmap e13: y(active, since 60s) 2024-03-24T05:28:11.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-24T05:28:11.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T05:28:11.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T05:28:11.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: mon.b (rank 2) addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] is down (out of quorum) 2024-03-24T05:28:11.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:11.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:11.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:11.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:10 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:11.295 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-24T05:28:11.295 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-24T05:28:11.546 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:28:11.995 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:11 smithi023 ceph-mon[25869]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:11.995 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:11 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:28:11.995 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:11 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:12.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:11 smithi143 ceph-mon[30288]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:12.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:11 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:28:12.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:11 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:13.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:13.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: mon.c calling monitor election 2024-03-24T05:28:13.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T05:28:13.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: mon.a calling monitor election 2024-03-24T05:28:13.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:13.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:28:13.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: from='client.? 172.21.15.143:0/3439166125' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T05:28:13.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:13.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:13.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:13.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:13.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:13.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:13.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:13.056 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T05:28:13.056 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0],c=[v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T05:28:13.056 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: fsmap 2024-03-24T05:28:13.056 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T05:28:13.056 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: mgrmap e13: y(active, since 60s) 2024-03-24T05:28:13.056 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-24T05:28:13.056 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T05:28:13.056 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T05:28:13.056 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: mon.b (rank 2) addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] is down (out of quorum) 2024-03-24T05:28:13.056 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.056 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.057 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.057 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.057 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:13.057 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:28:13.057 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:12 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:13.545 INFO:teuthology.orchestra.run.smithi023.stdout:# minimal ceph.conf for e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:28:13.545 INFO:teuthology.orchestra.run.smithi023.stdout:[global] 2024-03-24T05:28:13.545 INFO:teuthology.orchestra.run.smithi023.stdout: fsid = e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:28:13.545 INFO:teuthology.orchestra.run.smithi023.stdout: mon_host = [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-03-24T05:28:13.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: mon.b calling monitor election 2024-03-24T05:28:13.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:13.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-24T05:28:13.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-24T05:28:13.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T05:28:13.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: Updating smithi063:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:28:13.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: Updating smithi023:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:28:13.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: Updating smithi143:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:28:13.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: mon.b calling monitor election 2024-03-24T05:28:13.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: mon.c calling monitor election 2024-03-24T05:28:13.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: mon.a calling monitor election 2024-03-24T05:28:13.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T05:28:13.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0],c=[v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T05:28:13.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: fsmap 2024-03-24T05:28:13.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T05:28:13.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: mgrmap e13: y(active, since 62s) 2024-03-24T05:28:13.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-24T05:28:13.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: Cluster is now healthy 2024-03-24T05:28:13.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: overall HEALTH_OK 2024-03-24T05:28:13.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T05:28:13.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T05:28:13.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T05:28:13.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:13.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:13 smithi023 ceph-mon[25869]: Reconfiguring daemon mon.a on smithi023 2024-03-24T05:28:13.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: mon.b calling monitor election 2024-03-24T05:28:13.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:13.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-24T05:28:13.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-24T05:28:13.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T05:28:13.555 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: Updating smithi063:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:28:13.555 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: Updating smithi023:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:28:13.555 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: Updating smithi143:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:28:13.555 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: mon.b calling monitor election 2024-03-24T05:28:13.555 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: mon.c calling monitor election 2024-03-24T05:28:13.555 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: mon.a calling monitor election 2024-03-24T05:28:13.555 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T05:28:13.555 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0],c=[v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T05:28:13.555 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: fsmap 2024-03-24T05:28:13.556 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T05:28:13.556 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: mgrmap e13: y(active, since 62s) 2024-03-24T05:28:13.556 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-24T05:28:13.556 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: Cluster is now healthy 2024-03-24T05:28:13.556 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: overall HEALTH_OK 2024-03-24T05:28:13.556 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.556 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.556 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.556 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.556 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.556 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.557 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.557 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.557 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.557 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.557 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.557 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T05:28:13.557 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T05:28:13.557 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T05:28:13.557 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:13.557 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:13 smithi063 ceph-mon[30595]: Reconfiguring daemon mon.a on smithi023 2024-03-24T05:28:13.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: mon.b calling monitor election 2024-03-24T05:28:13.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:13.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-24T05:28:13.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-24T05:28:13.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T05:28:13.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: Updating smithi063:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:28:13.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: Updating smithi023:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:28:13.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: Updating smithi143:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:28:13.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: mon.b calling monitor election 2024-03-24T05:28:13.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: mon.c calling monitor election 2024-03-24T05:28:13.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: mon.a calling monitor election 2024-03-24T05:28:13.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T05:28:13.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0],c=[v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T05:28:13.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: fsmap 2024-03-24T05:28:13.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T05:28:13.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: mgrmap e13: y(active, since 62s) 2024-03-24T05:28:13.560 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-24T05:28:13.560 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: Cluster is now healthy 2024-03-24T05:28:13.560 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: overall HEALTH_OK 2024-03-24T05:28:13.560 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.560 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.560 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.560 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.560 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.560 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.560 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.560 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.561 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.561 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.561 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:13.561 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T05:28:13.561 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T05:28:13.561 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T05:28:13.561 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:13.561 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:13 smithi143 ceph-mon[30288]: Reconfiguring daemon mon.a on smithi023 2024-03-24T05:28:14.132 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-24T05:28:14.133 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T05:28:14.133 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T05:28:14.161 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T05:28:14.161 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T05:28:14.228 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-24T05:28:14.229 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T05:28:14.263 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-24T05:28:14.263 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T05:28:14.333 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T05:28:14.333 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T05:28:14.368 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T05:28:14.368 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T05:28:14.437 INFO:tasks.cephadm:Adding mgr.y on smithi023 2024-03-24T05:28:14.438 INFO:tasks.cephadm:Adding mgr.x on smithi063 2024-03-24T05:28:14.438 INFO:tasks.cephadm:Adding mgr.z on smithi143 2024-03-24T05:28:14.438 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph orch apply mgr '3;smithi023=y;smithi063=x;smithi143=z' 2024-03-24T05:28:14.553 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:14 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/3213134083' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:14.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:14 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:14.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:14 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/3213134083' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:14.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:14 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:14.709 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T05:28:14.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:14 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/3213134083' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:14.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:14 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:28:16.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:15 smithi063 ceph-mon[30595]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:16.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:15 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:16.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:15 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:16.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:15 smithi063 ceph-mon[30595]: Reconfiguring mon.b (monmap changed)... 2024-03-24T05:28:16.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:15 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T05:28:16.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:15 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T05:28:16.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:15 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:16.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:15 smithi063 ceph-mon[30595]: Reconfiguring daemon mon.b on smithi063 2024-03-24T05:28:16.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:15 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:16.174 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:15 smithi143 ceph-mon[30288]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:16.174 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:15 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:16.174 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:15 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:16.174 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:15 smithi143 ceph-mon[30288]: Reconfiguring mon.b (monmap changed)... 2024-03-24T05:28:16.174 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:15 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T05:28:16.175 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:15 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T05:28:16.175 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:15 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:16.175 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:15 smithi143 ceph-mon[30288]: Reconfiguring daemon mon.b on smithi063 2024-03-24T05:28:16.175 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:15 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:16.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:15 smithi023 ceph-mon[25869]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:16.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:15 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:16.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:15 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:16.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:15 smithi023 ceph-mon[25869]: Reconfiguring mon.b (monmap changed)... 2024-03-24T05:28:16.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:15 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T05:28:16.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:15 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T05:28:16.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:15 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:16.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:15 smithi023 ceph-mon[25869]: Reconfiguring daemon mon.b on smithi063 2024-03-24T05:28:16.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:15 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:16.426 INFO:teuthology.orchestra.run.smithi143.stdout:Scheduled mgr update... 2024-03-24T05:28:17.003 DEBUG:teuthology.orchestra.run.smithi063:mgr.x> sudo journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mgr.x.service 2024-03-24T05:28:17.007 DEBUG:teuthology.orchestra.run.smithi143:mgr.z> sudo journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mgr.z.service 2024-03-24T05:28:17.009 INFO:tasks.cephadm:Deploying OSDs... 2024-03-24T05:28:17.009 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T05:28:17.010 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T05:28:17.030 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-24T05:28:17.030 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/nvme1n1 2024-03-24T05:28:17.088 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/nvme1n1 2024-03-24T05:28:17.088 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T05:28:17.088 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-24T05:28:17.088 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T05:28:17.088 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T05:28:17.088 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-24 05:27:20.586133218 +0000 2024-03-24T05:28:17.088 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-24 05:25:07.176242477 +0000 2024-03-24T05:28:17.088 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-24 05:25:07.176242477 +0000 2024-03-24T05:28:17.088 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-24 05:25:07.163242581 +0000 2024-03-24T05:28:17.089 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-24T05:28:17.158 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-24T05:28:17.159 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-24T05:28:17.159 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000328198 s, 1.6 MB/s 2024-03-24T05:28:17.160 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-24T05:28:17.218 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/nvme2n1 2024-03-24T05:28:17.275 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/nvme2n1 2024-03-24T05:28:17.275 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T05:28:17.275 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-24T05:28:17.275 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T05:28:17.275 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T05:28:17.275 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-24 05:27:20.598133122 +0000 2024-03-24T05:28:17.276 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-24 05:25:07.331241237 +0000 2024-03-24T05:28:17.276 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-24 05:25:07.331241237 +0000 2024-03-24T05:28:17.276 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-24 05:25:07.323241301 +0000 2024-03-24T05:28:17.276 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-24T05:28:17.347 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-24T05:28:17.347 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-24T05:28:17.347 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000292184 s, 1.8 MB/s 2024-03-24T05:28:17.347 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-24T05:28:17.404 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/nvme3n1 2024-03-24T05:28:17.460 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/nvme3n1 2024-03-24T05:28:17.460 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T05:28:17.460 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-24T05:28:17.460 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T05:28:17.460 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T05:28:17.460 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-24 05:27:20.609133034 +0000 2024-03-24T05:28:17.460 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-24 05:25:07.481240037 +0000 2024-03-24T05:28:17.460 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-24 05:25:07.481240037 +0000 2024-03-24T05:28:17.460 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-24 05:25:07.473240101 +0000 2024-03-24T05:28:17.461 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-24T05:28:17.489 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:17 smithi143 ceph-mon[30288]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:17.489 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:17 smithi143 ceph-mon[30288]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=y;smithi063=x;smithi143=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:28:17.489 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:17 smithi143 ceph-mon[30288]: Saving service mgr spec with placement smithi023=y;smithi063=x;smithi143=z;count:3 2024-03-24T05:28:17.489 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:17 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:17.489 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:17 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:17.489 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:17 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:17.489 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:17 smithi143 ceph-mon[30288]: Reconfiguring mon.c (monmap changed)... 2024-03-24T05:28:17.489 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:17 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T05:28:17.489 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:17 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T05:28:17.490 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:17 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:17.490 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:17 smithi143 ceph-mon[30288]: Reconfiguring daemon mon.c on smithi143 2024-03-24T05:28:17.526 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-24T05:28:17.526 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-24T05:28:17.526 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000305284 s, 1.7 MB/s 2024-03-24T05:28:17.528 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-24T05:28:17.584 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/nvme4n1 2024-03-24T05:28:17.639 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/nvme4n1 2024-03-24T05:28:17.639 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T05:28:17.639 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 800 Links: 1 Device type: 103,8 2024-03-24T05:28:17.639 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T05:28:17.639 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T05:28:17.640 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-24 05:27:20.619132954 +0000 2024-03-24T05:28:17.640 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-24 05:25:07.637234472 +0000 2024-03-24T05:28:17.640 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-24 05:25:07.637234472 +0000 2024-03-24T05:28:17.640 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-24 05:25:07.629234862 +0000 2024-03-24T05:28:17.640 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-24T05:28:17.703 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:17 smithi023 ceph-mon[25869]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:17.703 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:17 smithi023 ceph-mon[25869]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=y;smithi063=x;smithi143=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:28:17.704 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:17 smithi023 ceph-mon[25869]: Saving service mgr spec with placement smithi023=y;smithi063=x;smithi143=z;count:3 2024-03-24T05:28:17.704 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:17 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:17.704 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:17 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:17.704 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:17 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:17.704 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:17 smithi023 ceph-mon[25869]: Reconfiguring mon.c (monmap changed)... 2024-03-24T05:28:17.704 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:17 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T05:28:17.704 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:17 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T05:28:17.704 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:17 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:17.704 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:17 smithi023 ceph-mon[25869]: Reconfiguring daemon mon.c on smithi143 2024-03-24T05:28:17.706 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-24T05:28:17.706 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-24T05:28:17.706 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000314593 s, 1.6 MB/s 2024-03-24T05:28:17.707 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-24T05:28:17.763 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-24T05:28:17.763 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T05:28:17.783 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-24T05:28:17.783 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/nvme1n1 2024-03-24T05:28:17.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:17 smithi063 ceph-mon[30595]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:17.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:17 smithi063 ceph-mon[30595]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=y;smithi063=x;smithi143=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:28:17.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:17 smithi063 ceph-mon[30595]: Saving service mgr spec with placement smithi023=y;smithi063=x;smithi143=z;count:3 2024-03-24T05:28:17.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:17 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:17.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:17 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:17.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:17 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:17.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:17 smithi063 ceph-mon[30595]: Reconfiguring mon.c (monmap changed)... 2024-03-24T05:28:17.805 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:17 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T05:28:17.805 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:17 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T05:28:17.805 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:17 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:17.805 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:17 smithi063 ceph-mon[30595]: Reconfiguring daemon mon.c on smithi143 2024-03-24T05:28:17.822 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/nvme1n1 2024-03-24T05:28:17.822 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T05:28:17.822 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 764 Links: 1 Device type: 103,2 2024-03-24T05:28:17.822 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T05:28:17.822 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T05:28:17.822 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-24 05:27:37.560985395 +0000 2024-03-24T05:28:17.822 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-24 05:25:08.829638712 +0000 2024-03-24T05:28:17.822 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-24 05:25:08.829638712 +0000 2024-03-24T05:28:17.823 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-24 05:25:08.821638801 +0000 2024-03-24T05:28:17.823 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-24T05:28:17.891 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-24T05:28:17.891 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-24T05:28:17.892 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.00036731 s, 1.4 MB/s 2024-03-24T05:28:17.893 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-24T05:28:17.951 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/nvme2n1 2024-03-24T05:28:18.007 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/nvme2n1 2024-03-24T05:28:18.007 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T05:28:18.007 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-03-24T05:28:18.008 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T05:28:18.008 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T05:28:18.008 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-24 05:27:37.571985273 +0000 2024-03-24T05:28:18.008 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-24 05:25:08.982637014 +0000 2024-03-24T05:28:18.008 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-24 05:25:08.982637014 +0000 2024-03-24T05:28:18.008 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-24 05:25:08.970637147 +0000 2024-03-24T05:28:18.008 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-24T05:28:18.075 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-24T05:28:18.075 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-24T05:28:18.075 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000226844 s, 2.3 MB/s 2024-03-24T05:28:18.076 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-24T05:28:18.133 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/nvme3n1 2024-03-24T05:28:18.189 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/nvme3n1 2024-03-24T05:28:18.189 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T05:28:18.189 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-24T05:28:18.189 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T05:28:18.189 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T05:28:18.189 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-24 05:27:37.582985152 +0000 2024-03-24T05:28:18.189 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-24 05:25:09.133635337 +0000 2024-03-24T05:28:18.190 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-24 05:25:09.133635337 +0000 2024-03-24T05:28:18.190 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-24 05:25:09.121635471 +0000 2024-03-24T05:28:18.190 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-24T05:28:18.255 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-24T05:28:18.255 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-24T05:28:18.255 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000253031 s, 2.0 MB/s 2024-03-24T05:28:18.257 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-24T05:28:18.313 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/nvme4n1 2024-03-24T05:28:18.368 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/nvme4n1 2024-03-24T05:28:18.368 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T05:28:18.368 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 798 Links: 1 Device type: 103,8 2024-03-24T05:28:18.368 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T05:28:18.369 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T05:28:18.369 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-24 05:27:37.592985041 +0000 2024-03-24T05:28:18.369 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-24 05:25:09.291633583 +0000 2024-03-24T05:28:18.369 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-24 05:25:09.291633583 +0000 2024-03-24T05:28:18.369 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-24 05:25:09.274633772 +0000 2024-03-24T05:28:18.369 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-24T05:28:18.433 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-24T05:28:18.434 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-24T05:28:18.434 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000210656 s, 2.4 MB/s 2024-03-24T05:28:18.435 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-24T05:28:18.491 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T05:28:18.491 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T05:28:18.511 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-24T05:28:18.511 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/nvme1n1 2024-03-24T05:28:18.570 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/nvme1n1 2024-03-24T05:28:18.570 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T05:28:18.570 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-24T05:28:18.570 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T05:28:18.570 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T05:28:18.570 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-24 05:27:46.763821654 +0000 2024-03-24T05:28:18.570 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-24 05:25:10.477001759 +0000 2024-03-24T05:28:18.570 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-24 05:25:10.477001759 +0000 2024-03-24T05:28:18.571 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-03-24 05:25:10.464002107 +0000 2024-03-24T05:28:18.571 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-24T05:28:18.640 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-24T05:28:18.640 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-24T05:28:18.641 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000254552 s, 2.0 MB/s 2024-03-24T05:28:18.642 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-24T05:28:18.701 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/nvme2n1 2024-03-24T05:28:18.759 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/nvme2n1 2024-03-24T05:28:18.759 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T05:28:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-24T05:28:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T05:28:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T05:28:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-24 05:27:46.774821360 +0000 2024-03-24T05:28:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-24 05:25:10.627997723 +0000 2024-03-24T05:28:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-24 05:25:10.627997723 +0000 2024-03-24T05:28:18.759 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-03-24 05:25:10.615998043 +0000 2024-03-24T05:28:18.760 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-24T05:28:18.827 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-24T05:28:18.827 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-24T05:28:18.827 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000207849 s, 2.5 MB/s 2024-03-24T05:28:18.828 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-24T05:28:18.887 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/nvme3n1 2024-03-24T05:28:18.943 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/nvme3n1 2024-03-24T05:28:18.943 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T05:28:18.943 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-24T05:28:18.943 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T05:28:18.943 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T05:28:18.943 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-24 05:27:46.785821067 +0000 2024-03-24T05:28:18.944 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-24 05:25:10.776993739 +0000 2024-03-24T05:28:18.944 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-24 05:25:10.776993739 +0000 2024-03-24T05:28:18.944 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-03-24 05:25:10.765994033 +0000 2024-03-24T05:28:18.944 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-24T05:28:19.008 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-24T05:28:19.009 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-24T05:28:19.009 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000246809 s, 2.1 MB/s 2024-03-24T05:28:19.010 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-24T05:28:19.065 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/nvme4n1 2024-03-24T05:28:19.121 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/nvme4n1 2024-03-24T05:28:19.121 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T05:28:19.122 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-24T05:28:19.122 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T05:28:19.122 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T05:28:19.122 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-24 05:27:46.796820773 +0000 2024-03-24T05:28:19.122 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-24 05:25:10.924989782 +0000 2024-03-24T05:28:19.122 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-24 05:25:10.924989782 +0000 2024-03-24T05:28:19.122 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-03-24 05:25:10.913990076 +0000 2024-03-24T05:28:19.122 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-24T05:28:19.187 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-24T05:28:19.187 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-24T05:28:19.187 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000242887 s, 2.1 MB/s 2024-03-24T05:28:19.189 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-24T05:28:19.245 INFO:tasks.cephadm:Deploying osd.0 on smithi023 with /dev/nvme4n1... 2024-03-24T05:28:19.245 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-24T05:28:19.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:19 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:19.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:19 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:19.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:19 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:28:19.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:19 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:19.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:19 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:28:19.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:19 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:19.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:19 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T05:28:19.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:19 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T05:28:19.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:19 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T05:28:19.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:19 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:19.483 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:28:19.553 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:19 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:19.553 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:19 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:19.553 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:19 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:28:19.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:19 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:19.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:19 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:28:19.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:19 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:19.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:19 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T05:28:19.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:19 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T05:28:19.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:19 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T05:28:19.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:19 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:19.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:19 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:19.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:19 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:19.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:19 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:28:19.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:19 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:19.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:19 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:28:19.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:19 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:19.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:19 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T05:28:19.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:19 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T05:28:19.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:19 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T05:28:19.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:19 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:20.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:20 smithi023 ceph-mon[25869]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:20.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:20 smithi023 ceph-mon[25869]: Deploying daemon mgr.z on smithi143 2024-03-24T05:28:20.331 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:20 smithi143 ceph-mon[30288]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:20.331 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:20 smithi143 ceph-mon[30288]: Deploying daemon mgr.z on smithi143 2024-03-24T05:28:20.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:20 smithi063 ceph-mon[30595]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:20.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:20 smithi063 ceph-mon[30595]: Deploying daemon mgr.z on smithi143 2024-03-24T05:28:20.984 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:20 smithi143 systemd[1]: Starting Ceph mgr.z for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:28:21.302 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:21 smithi143 ceph-mon[30288]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:21.303 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:20 smithi143 podman[32341]: 2024-03-24 05:28:20.984758216 +0000 UTC m=+0.105319468 container create 47dd699bae4bccc00e44ffa7d308c0dd1cebc7b6c3924f69c29fb30b06b15433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-24T05:28:21.303 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:20 smithi143 podman[32341]: 2024-03-24 05:28:20.896779111 +0000 UTC m=+0.017340325 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T05:28:21.303 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:21 smithi143 podman[32341]: 2024-03-24 05:28:21.195511941 +0000 UTC m=+0.316073157 container init 47dd699bae4bccc00e44ffa7d308c0dd1cebc7b6c3924f69c29fb30b06b15433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T05:28:21.303 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:21 smithi143 podman[32341]: 2024-03-24 05:28:21.201547457 +0000 UTC m=+0.322108662 container start 47dd699bae4bccc00e44ffa7d308c0dd1cebc7b6c3924f69c29fb30b06b15433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T05:28:21.303 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:21 smithi143 ceph-mgr[32380]: -- 172.21.15.143:0/3425618181 <== mon.1 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5627cfe661a0 con 0x5627cfe3cc00 2024-03-24T05:28:21.303 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:21 smithi143 bash[32341]: 47dd699bae4bccc00e44ffa7d308c0dd1cebc7b6c3924f69c29fb30b06b15433 2024-03-24T05:28:21.306 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:21 smithi023 ceph-mon[25869]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:21.553 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:21 smithi063 ceph-mon[30595]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:21.557 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:21 smithi143 systemd[1]: Started Ceph mgr.z for e2aa55c2-e99e-11ee-95cd-87774f69a715. 2024-03-24T05:28:21.752 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/nvme4n1 2024-03-24T05:28:21.752 INFO:teuthology.orchestra.run.smithi023.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-24T05:28:21.752 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-24T05:28:21.752 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-24T05:28:21.752 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-24T05:28:21.752 INFO:teuthology.orchestra.run.smithi023.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00896099 s, 1.2 GB/s 2024-03-24T05:28:21.752 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-24T05:28:22.057 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:21 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:21.788+0000 7f7280d2b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T05:28:22.058 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:21 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:21.881+0000 7f7280d2b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T05:28:22.378 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi023:/dev/nvme4n1 2024-03-24T05:28:22.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:22 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:22.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:22 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:22.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:22 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:22.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:22 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T05:28:22.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:22 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T05:28:22.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:22 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T05:28:22.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:22 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:22.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:22 smithi143 ceph-mon[30288]: Deploying daemon mgr.x on smithi063 2024-03-24T05:28:22.615 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:28:22.733 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:22 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:22.734 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:22 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:22.734 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:22 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:22.734 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:22 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T05:28:22.734 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:22 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T05:28:22.734 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:22 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T05:28:22.734 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:22 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:22.734 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:22 smithi063 ceph-mon[30595]: Deploying daemon mgr.x on smithi063 2024-03-24T05:28:22.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:22 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:22.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:22 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:22.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:22 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:22.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:22 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T05:28:22.737 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:22 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T05:28:22.737 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:22 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T05:28:22.737 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:22 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:22.737 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:22 smithi023 ceph-mon[25869]: Deploying daemon mgr.x on smithi063 2024-03-24T05:28:22.807 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:22 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:22.520+0000 7f7280d2b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T05:28:22.808 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:22 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:22.611+0000 7f7280d2b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T05:28:22.808 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:22 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:22.751+0000 7f7280d2b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T05:28:23.188 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:22 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:22.846+0000 7f7280d2b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T05:28:23.188 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:22 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:22.927+0000 7f7280d2b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T05:28:23.188 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:23 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:23.187+0000 7f7280d2b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T05:28:23.472 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:23 smithi143 ceph-mon[30288]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:23.473 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:23 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:23.296+0000 7f7280d2b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T05:28:23.473 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:23 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:23.380+0000 7f7280d2b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T05:28:23.473 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:23 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:23.473+0000 7f7280d2b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T05:28:23.553 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:23 smithi063 ceph-mon[30595]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:23.554 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:23 smithi063 systemd[1]: Starting Ceph mgr.x for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:28:23.789 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:23 smithi023 ceph-mon[25869]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:23.807 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:23 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:23.648+0000 7f7280d2b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T05:28:23.951 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:23 smithi063 podman[32037]: 2024-03-24 05:28:23.695252188 +0000 UTC m=+0.112485316 container create d03b804431107a5ebd5b6da9cbfa74e1a28dafaf7f869d0505e5c865d0f44514 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_CLEAN=True) 2024-03-24T05:28:23.951 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:23 smithi063 podman[32037]: 2024-03-24 05:28:23.600563964 +0000 UTC m=+0.017797110 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T05:28:23.951 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:23 smithi063 podman[32037]: 2024-03-24 05:28:23.922720114 +0000 UTC m=+0.339953257 container init d03b804431107a5ebd5b6da9cbfa74e1a28dafaf7f869d0505e5c865d0f44514 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T05:28:23.952 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:23 smithi063 podman[32037]: 2024-03-24 05:28:23.928302301 +0000 UTC m=+0.345535420 container start d03b804431107a5ebd5b6da9cbfa74e1a28dafaf7f869d0505e5c865d0f44514 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-db0330b) 2024-03-24T05:28:23.952 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:23 smithi063 ceph-mgr[32075]: -- 172.21.15.63:0/2855457001 <== mon.2 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559e1aa4e1a0 con 0x559e1aa24c00 2024-03-24T05:28:24.284 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:24 smithi063 bash[32037]: d03b804431107a5ebd5b6da9cbfa74e1a28dafaf7f869d0505e5c865d0f44514 2024-03-24T05:28:24.284 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:24 smithi063 systemd[1]: Started Ceph mgr.x for e2aa55c2-e99e-11ee-95cd-87774f69a715. 2024-03-24T05:28:24.554 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:24 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:24.503+0000 7fb7f142d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T05:28:25.053 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:24 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:24.603+0000 7fb7f142d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T05:28:25.057 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:24 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:24.630+0000 7f7280d2b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T05:28:25.058 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:24 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:24.770+0000 7f7280d2b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T05:28:25.317 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:25 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:25.317 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:25 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:25.317 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:25 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:25.317 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:25 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:25.317 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:25 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:28:25.317 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:25 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T05:28:25.317 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:25 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T05:28:25.317 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:25 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:25.405 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:25 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:25.405 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:25 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:25.405 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:25 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:25.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:25 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:25.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:25 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:28:25.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:25 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T05:28:25.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:25 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T05:28:25.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:25 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:25.406 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:25 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:25.117+0000 7f7280d2b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T05:28:25.406 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:25 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:25.211+0000 7f7280d2b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T05:28:25.406 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:25 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:25.405+0000 7f7280d2b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T05:28:25.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:25 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:25.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:25 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:25.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:25 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:25.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:25 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:25.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:25 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:28:25.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:25 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T05:28:25.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:25 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T05:28:25.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:25 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:25.603 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:25 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:25.352+0000 7fb7f142d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T05:28:25.604 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:25 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:25.452+0000 7fb7f142d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T05:28:25.807 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:25 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:25.596+0000 7f7280d2b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T05:28:26.053 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:25 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:25.604+0000 7fb7f142d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T05:28:26.053 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:25 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:25.702+0000 7fb7f142d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T05:28:26.053 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:25 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:25.786+0000 7fb7f142d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T05:28:26.053 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:26 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:26.053+0000 7fb7f142d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T05:28:26.135 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:25 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:25.877+0000 7f7280d2b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T05:28:26.135 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:26 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:26.136+0000 7f7280d2b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T05:28:26.304 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:26 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:26.165+0000 7fb7f142d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T05:28:26.304 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:26 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:26.253+0000 7fb7f142d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T05:28:26.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:26 smithi063 ceph-mon[30595]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:26.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:26 smithi063 ceph-mon[30595]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:28:26.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:26 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:26.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:26 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:26.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:26 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:26.305 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:26 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:26.305 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:26 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:26.305 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:26 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:26.305 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:26 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:28:26.305 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:26 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:26.305 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:26 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T05:28:26.305 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:26 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T05:28:26.305 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:26 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:26.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:26 smithi023 ceph-mon[25869]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:26.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:26 smithi023 ceph-mon[25869]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:28:26.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:26 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:26.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:26 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:26.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:26 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:26.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:26 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:26.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:26 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:26.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:26 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:26.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:26 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:28:26.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:26 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:26.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:26 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T05:28:26.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:26 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T05:28:26.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:26 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:26.557 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:26 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:26.367+0000 7f7280d2b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T05:28:26.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:26 smithi143 ceph-mon[30288]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:26.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:26 smithi143 ceph-mon[30288]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:28:26.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:26 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:26.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:26 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:26.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:26 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:26.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:26 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:26.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:26 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:26.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:26 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:26.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:26 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:28:26.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:26 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:26.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:26 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T05:28:26.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:26 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T05:28:26.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:26 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:26.803 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:26 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:26.349+0000 7fb7f142d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T05:28:26.804 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:26 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:26.526+0000 7fb7f142d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T05:28:27.524 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:27 smithi063 ceph-mon[30595]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T05:28:27.524 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:27 smithi063 ceph-mon[30595]: Reconfiguring daemon mgr.y on smithi023 2024-03-24T05:28:27.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:27 smithi143 ceph-mon[30288]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T05:28:27.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:27 smithi143 ceph-mon[30288]: Reconfiguring daemon mgr.y on smithi023 2024-03-24T05:28:27.558 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:27 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:27.147+0000 7f7280d2b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T05:28:27.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:27 smithi023 ceph-mon[25869]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T05:28:27.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:27 smithi023 ceph-mon[25869]: Reconfiguring daemon mgr.y on smithi023 2024-03-24T05:28:27.803 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:27 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:27.524+0000 7fb7f142d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T05:28:27.803 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:27 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:27.659+0000 7fb7f142d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T05:28:28.019 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:27 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:27.714+0000 7f7280d2b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T05:28:28.019 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:27 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:27.810+0000 7f7280d2b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T05:28:28.019 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:28:28 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:28:28.019+0000 7f7280d2b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T05:28:28.283 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:27 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:27.994+0000 7fb7f142d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T05:28:28.284 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:28 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:28.085+0000 7fb7f142d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T05:28:28.284 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:28 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:28.284+0000 7fb7f142d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T05:28:28.284 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:28 smithi063 ceph-mon[30595]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:28.284 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:28 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/3389274363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38483f5a-088a-4caa-9228-5502305de04d"}]: dispatch 2024-03-24T05:28:28.284 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:28 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/3389274363' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38483f5a-088a-4caa-9228-5502305de04d"}]': finished 2024-03-24T05:28:28.284 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:28 smithi063 ceph-mon[30595]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T05:28:28.284 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:28 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T05:28:28.284 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:28 smithi063 ceph-mon[30595]: Standby manager daemon z started 2024-03-24T05:28:28.284 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:28 smithi063 ceph-mon[30595]: from='mgr.? 172.21.15.143:0/2897486653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T05:28:28.284 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:28 smithi063 ceph-mon[30595]: from='mgr.? 172.21.15.143:0/2897486653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T05:28:28.284 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:28 smithi063 ceph-mon[30595]: from='mgr.? 172.21.15.143:0/2897486653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T05:28:28.284 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:28 smithi063 ceph-mon[30595]: from='mgr.? 172.21.15.143:0/2897486653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T05:28:28.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:28 smithi143 ceph-mon[30288]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:28.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:28 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/3389274363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38483f5a-088a-4caa-9228-5502305de04d"}]: dispatch 2024-03-24T05:28:28.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:28 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/3389274363' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38483f5a-088a-4caa-9228-5502305de04d"}]': finished 2024-03-24T05:28:28.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:28 smithi143 ceph-mon[30288]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T05:28:28.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:28 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T05:28:28.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:28 smithi143 ceph-mon[30288]: Standby manager daemon z started 2024-03-24T05:28:28.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:28 smithi143 ceph-mon[30288]: from='mgr.? 172.21.15.143:0/2897486653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T05:28:28.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:28 smithi143 ceph-mon[30288]: from='mgr.? 172.21.15.143:0/2897486653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T05:28:28.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:28 smithi143 ceph-mon[30288]: from='mgr.? 172.21.15.143:0/2897486653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T05:28:28.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:28 smithi143 ceph-mon[30288]: from='mgr.? 172.21.15.143:0/2897486653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T05:28:28.553 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:28 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:28.470+0000 7fb7f142d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T05:28:28.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:28 smithi023 ceph-mon[25869]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:28.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:28 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/3389274363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38483f5a-088a-4caa-9228-5502305de04d"}]: dispatch 2024-03-24T05:28:28.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:28 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/3389274363' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38483f5a-088a-4caa-9228-5502305de04d"}]': finished 2024-03-24T05:28:28.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:28 smithi023 ceph-mon[25869]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T05:28:28.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:28 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T05:28:28.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:28 smithi023 ceph-mon[25869]: Standby manager daemon z started 2024-03-24T05:28:28.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:28 smithi023 ceph-mon[25869]: from='mgr.? 172.21.15.143:0/2897486653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T05:28:28.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:28 smithi023 ceph-mon[25869]: from='mgr.? 172.21.15.143:0/2897486653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T05:28:28.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:28 smithi023 ceph-mon[25869]: from='mgr.? 172.21.15.143:0/2897486653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T05:28:28.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:28 smithi023 ceph-mon[25869]: from='mgr.? 172.21.15.143:0/2897486653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T05:28:29.019 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:28 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:28.759+0000 7fb7f142d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T05:28:29.303 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:29 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:29.020+0000 7fb7f142d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T05:28:29.303 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:29 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:29.259+0000 7fb7f142d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T05:28:29.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:29 smithi063 ceph-mon[30595]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:29.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:29 smithi063 ceph-mon[30595]: mgrmap e14: y(active, since 78s), standbys: z 2024-03-24T05:28:29.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:29 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T05:28:29.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:29 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/801199427' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T05:28:29.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:29 smithi143 ceph-mon[30288]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:29.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:29 smithi143 ceph-mon[30288]: mgrmap e14: y(active, since 78s), standbys: z 2024-03-24T05:28:29.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:29 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T05:28:29.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:29 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/801199427' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T05:28:29.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:29 smithi023 ceph-mon[25869]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:29.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:29 smithi023 ceph-mon[25869]: mgrmap e14: y(active, since 78s), standbys: z 2024-03-24T05:28:29.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:29 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T05:28:29.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:29 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/801199427' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T05:28:30.553 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:30 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:30.081+0000 7fb7f142d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T05:28:30.980 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:30 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:30.661+0000 7fb7f142d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T05:28:30.980 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:30 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:30.760+0000 7fb7f142d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T05:28:30.980 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:28:30 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:28:30.980+0000 7fb7f142d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T05:28:31.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:31 smithi023 ceph-mon[25869]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:31.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:31 smithi023 ceph-mon[25869]: Standby manager daemon x started 2024-03-24T05:28:31.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:31 smithi023 ceph-mon[25869]: from='mgr.? 172.21.15.63:0/2073247429' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T05:28:31.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:31 smithi023 ceph-mon[25869]: from='mgr.? 172.21.15.63:0/2073247429' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T05:28:31.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:31 smithi023 ceph-mon[25869]: from='mgr.? 172.21.15.63:0/2073247429' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T05:28:31.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:31 smithi023 ceph-mon[25869]: from='mgr.? 172.21.15.63:0/2073247429' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T05:28:31.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:31 smithi063 ceph-mon[30595]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:31.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:31 smithi063 ceph-mon[30595]: Standby manager daemon x started 2024-03-24T05:28:31.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:31 smithi063 ceph-mon[30595]: from='mgr.? 172.21.15.63:0/2073247429' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T05:28:31.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:31 smithi063 ceph-mon[30595]: from='mgr.? 172.21.15.63:0/2073247429' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T05:28:31.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:31 smithi063 ceph-mon[30595]: from='mgr.? 172.21.15.63:0/2073247429' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T05:28:31.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:31 smithi063 ceph-mon[30595]: from='mgr.? 172.21.15.63:0/2073247429' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T05:28:31.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:31 smithi143 ceph-mon[30288]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:31.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:31 smithi143 ceph-mon[30288]: Standby manager daemon x started 2024-03-24T05:28:31.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:31 smithi143 ceph-mon[30288]: from='mgr.? 172.21.15.63:0/2073247429' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T05:28:31.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:31 smithi143 ceph-mon[30288]: from='mgr.? 172.21.15.63:0/2073247429' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T05:28:31.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:31 smithi143 ceph-mon[30288]: from='mgr.? 172.21.15.63:0/2073247429' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T05:28:31.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:31 smithi143 ceph-mon[30288]: from='mgr.? 172.21.15.63:0/2073247429' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T05:28:32.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:32 smithi063 ceph-mon[30595]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-24T05:28:32.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:32 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T05:28:32.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:32 smithi143 ceph-mon[30288]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-24T05:28:32.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:32 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T05:28:32.819 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:32 smithi023 ceph-mon[25869]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-24T05:28:32.819 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:32 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T05:28:33.705 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:33 smithi023 ceph-mon[25869]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:33.706 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:33 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:33.706 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:33 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:33.706 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:33 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:28:33.706 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:33 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:33.706 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:33 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:28:33.706 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:33 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:34.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:33 smithi063 ceph-mon[30595]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:34.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:33 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:34.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:33 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:34.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:33 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:28:34.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:33 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:34.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:33 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:28:34.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:33 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:34.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:33 smithi143 ceph-mon[30288]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:34.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:33 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:34.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:33 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:34.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:33 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:28:34.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:33 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:34.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:33 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:28:34.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:33 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:35.912 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:35 smithi023 ceph-mon[25869]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:35.912 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:35 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T05:28:35.912 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:35 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:35.912 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:35 smithi023 ceph-mon[25869]: Deploying daemon osd.0 on smithi023 2024-03-24T05:28:36.053 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:35 smithi063 ceph-mon[30595]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:36.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:35 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T05:28:36.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:35 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:36.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:35 smithi063 ceph-mon[30595]: Deploying daemon osd.0 on smithi023 2024-03-24T05:28:36.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:35 smithi143 ceph-mon[30288]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:36.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:35 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T05:28:36.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:35 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:36.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:35 smithi143 ceph-mon[30288]: Deploying daemon osd.0 on smithi023 2024-03-24T05:28:37.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:37 smithi023 ceph-mon[25869]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:38.053 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:37 smithi063 ceph-mon[30595]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:38.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:37 smithi143 ceph-mon[30288]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:40.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:39 smithi063 ceph-mon[30595]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:40.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:39 smithi143 ceph-mon[30288]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:40.073 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:39 smithi023 ceph-mon[25869]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:41.689 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:41 smithi023 ceph-mon[25869]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:41.690 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:41 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:41.690 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:41 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:41.690 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:41 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:28:41.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:41 smithi063 ceph-mon[30595]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:41.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:41 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:41.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:41 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:41.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:41 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:28:41.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:41 smithi143 ceph-mon[30288]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:41.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:41 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:41.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:41 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:41.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:41 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:28:43.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:43 smithi023 ceph-mon[25869]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:43.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:43 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:43.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:43 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:43.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:43 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:43.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:43 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:28:43.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:43 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:43.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:43 smithi063 ceph-mon[30595]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:43.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:43 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:43.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:43 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:43.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:43 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:43.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:43 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:28:43.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:43 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:43.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:43 smithi143 ceph-mon[30288]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:43.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:43 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:43.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:43 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:43.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:43 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:43.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:43 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:28:43.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:43 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:43.920 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 0 on host 'smithi023' 2024-03-24T05:28:44.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:44 smithi023 ceph-mon[25869]: from='osd.0 [v2:172.21.15.23:6802/489464929,v1:172.21.15.23:6803/489464929]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T05:28:44.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:44 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:44.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:44 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:44.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:44 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:28:44.562 DEBUG:teuthology.orchestra.run.smithi023:osd.0> sudo journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@osd.0.service 2024-03-24T05:28:44.565 INFO:tasks.cephadm:Deploying osd.1 on smithi063 with /dev/nvme4n1... 2024-03-24T05:28:44.565 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-24T05:28:44.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:44 smithi063 ceph-mon[30595]: from='osd.0 [v2:172.21.15.23:6802/489464929,v1:172.21.15.23:6803/489464929]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T05:28:44.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:44 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:44.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:44 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:44.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:44 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:28:44.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:44 smithi143 ceph-mon[30288]: from='osd.0 [v2:172.21.15.23:6802/489464929,v1:172.21.15.23:6803/489464929]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T05:28:44.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:44 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:44.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:44 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:44.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:44 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:28:44.811 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T05:28:45.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:45 smithi063 ceph-mon[30595]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:45.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:45 smithi063 ceph-mon[30595]: from='osd.0 [v2:172.21.15.23:6802/489464929,v1:172.21.15.23:6803/489464929]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T05:28:45.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:45 smithi063 ceph-mon[30595]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T05:28:45.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:45 smithi063 ceph-mon[30595]: from='osd.0 [v2:172.21.15.23:6802/489464929,v1:172.21.15.23:6803/489464929]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-24T05:28:45.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:45 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T05:28:45.693 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:45 smithi023 ceph-mon[25869]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:45.693 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:45 smithi023 ceph-mon[25869]: from='osd.0 [v2:172.21.15.23:6802/489464929,v1:172.21.15.23:6803/489464929]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T05:28:45.693 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:45 smithi023 ceph-mon[25869]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T05:28:45.693 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:45 smithi023 ceph-mon[25869]: from='osd.0 [v2:172.21.15.23:6802/489464929,v1:172.21.15.23:6803/489464929]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-24T05:28:45.693 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:45 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T05:28:45.694 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:28:45 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-0[40745]: 2024-03-24T05:28:45.396+0000 7ff592aa5700 -1 osd.0 0 waiting for initial osdmap 2024-03-24T05:28:45.694 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:28:45 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-0[40745]: 2024-03-24T05:28:45.401+0000 7ff58e898700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T05:28:45.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:45 smithi143 ceph-mon[30288]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:45.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:45 smithi143 ceph-mon[30288]: from='osd.0 [v2:172.21.15.23:6802/489464929,v1:172.21.15.23:6803/489464929]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T05:28:45.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:45 smithi143 ceph-mon[30288]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T05:28:45.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:45 smithi143 ceph-mon[30288]: from='osd.0 [v2:172.21.15.23:6802/489464929,v1:172.21.15.23:6803/489464929]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-24T05:28:45.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:45 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T05:28:46.687 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/nvme4n1 2024-03-24T05:28:46.687 INFO:teuthology.orchestra.run.smithi063.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-24T05:28:46.687 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-24T05:28:46.687 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-03-24T05:28:46.687 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-03-24T05:28:46.687 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0141179 s, 743 MB/s 2024-03-24T05:28:46.688 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-03-24T05:28:46.688 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:46 smithi063 ceph-mon[30595]: from='osd.0 [v2:172.21.15.23:6802/489464929,v1:172.21.15.23:6803/489464929]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-24T05:28:46.688 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:46 smithi063 ceph-mon[30595]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T05:28:46.688 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:46 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T05:28:46.688 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:46 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T05:28:46.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:46 smithi143 ceph-mon[30288]: from='osd.0 [v2:172.21.15.23:6802/489464929,v1:172.21.15.23:6803/489464929]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-24T05:28:46.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:46 smithi143 ceph-mon[30288]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T05:28:46.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:46 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T05:28:46.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:46 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T05:28:46.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:46 smithi023 ceph-mon[25869]: from='osd.0 [v2:172.21.15.23:6802/489464929,v1:172.21.15.23:6803/489464929]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-24T05:28:46.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:46 smithi023 ceph-mon[25869]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T05:28:46.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:46 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T05:28:46.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:46 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T05:28:47.379 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi063:/dev/nvme4n1 2024-03-24T05:28:47.619 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T05:28:47.724 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:47 smithi023 ceph-mon[25869]: purged_snaps scrub starts 2024-03-24T05:28:47.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:47 smithi023 ceph-mon[25869]: purged_snaps scrub ok 2024-03-24T05:28:47.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:47 smithi023 ceph-mon[25869]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:47.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:47 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T05:28:47.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:47 smithi023 ceph-mon[25869]: osd.0 [v2:172.21.15.23:6802/489464929,v1:172.21.15.23:6803/489464929] boot 2024-03-24T05:28:47.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:47 smithi023 ceph-mon[25869]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T05:28:47.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:47 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T05:28:47.728 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:47 smithi063 ceph-mon[30595]: purged_snaps scrub starts 2024-03-24T05:28:47.728 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:47 smithi063 ceph-mon[30595]: purged_snaps scrub ok 2024-03-24T05:28:47.728 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:47 smithi063 ceph-mon[30595]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:47.728 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:47 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T05:28:47.728 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:47 smithi063 ceph-mon[30595]: osd.0 [v2:172.21.15.23:6802/489464929,v1:172.21.15.23:6803/489464929] boot 2024-03-24T05:28:47.728 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:47 smithi063 ceph-mon[30595]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T05:28:47.728 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:47 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T05:28:47.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:47 smithi143 ceph-mon[30288]: purged_snaps scrub starts 2024-03-24T05:28:47.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:47 smithi143 ceph-mon[30288]: purged_snaps scrub ok 2024-03-24T05:28:47.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:47 smithi143 ceph-mon[30288]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T05:28:47.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:47 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T05:28:47.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:47 smithi143 ceph-mon[30288]: osd.0 [v2:172.21.15.23:6802/489464929,v1:172.21.15.23:6803/489464929] boot 2024-03-24T05:28:47.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:47 smithi143 ceph-mon[30288]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T05:28:47.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:47 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T05:28:48.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:48 smithi063 ceph-mon[30595]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T05:28:48.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:48 smithi063 ceph-mon[30595]: Detected new or changed devices on smithi023 2024-03-24T05:28:48.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:48 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:48.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:48 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:48.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:48 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T05:28:48.805 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:48 smithi063 ceph-mon[30595]: Adjusting osd_memory_target on smithi023 to 17088M 2024-03-24T05:28:48.805 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:48 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:48.805 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:48 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:48.805 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:48 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:28:48.805 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:48 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:48.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:48 smithi143 ceph-mon[30288]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T05:28:48.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:48 smithi143 ceph-mon[30288]: Detected new or changed devices on smithi023 2024-03-24T05:28:48.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:48 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:48.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:48 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:48.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:48 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T05:28:48.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:48 smithi143 ceph-mon[30288]: Adjusting osd_memory_target on smithi023 to 17088M 2024-03-24T05:28:48.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:48 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:48.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:48 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:48.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:48 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:28:48.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:48 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:48.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:48 smithi023 ceph-mon[25869]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T05:28:48.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:48 smithi023 ceph-mon[25869]: Detected new or changed devices on smithi023 2024-03-24T05:28:48.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:48 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:48.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:48 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:48.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:48 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T05:28:48.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:48 smithi023 ceph-mon[25869]: Adjusting osd_memory_target on smithi023 to 17088M 2024-03-24T05:28:48.831 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:48 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:48.831 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:48 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:48.831 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:48 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:28:48.831 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:48 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:28:50.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:49 smithi063 ceph-mon[30595]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:28:50.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:49 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T05:28:50.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:49 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T05:28:50.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:49 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:50.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:49 smithi143 ceph-mon[30288]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:28:50.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:49 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T05:28:50.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:49 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T05:28:50.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:49 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:50.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:49 smithi023 ceph-mon[25869]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:28:50.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:49 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T05:28:50.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:49 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T05:28:50.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:49 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:50.967 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:50 smithi063 ceph-mon[30595]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:28:51.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:50 smithi143 ceph-mon[30288]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:28:51.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:50 smithi023 ceph-mon[25869]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:28:52.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:51 smithi063 ceph-mon[30595]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:28:52.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:51 smithi143 ceph-mon[30288]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:28:52.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:51 smithi023 ceph-mon[25869]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:28:53.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:52 smithi143 ceph-mon[30288]: from='client.? 172.21.15.63:0/2416552611' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb781706-c5f9-4f47-8f0b-5784a95159a4"}]: dispatch 2024-03-24T05:28:53.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:52 smithi143 ceph-mon[30288]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb781706-c5f9-4f47-8f0b-5784a95159a4"}]: dispatch 2024-03-24T05:28:53.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:52 smithi143 ceph-mon[30288]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb781706-c5f9-4f47-8f0b-5784a95159a4"}]': finished 2024-03-24T05:28:53.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:52 smithi143 ceph-mon[30288]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T05:28:53.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:52 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T05:28:53.073 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:52 smithi063 ceph-mon[30595]: from='client.? 172.21.15.63:0/2416552611' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb781706-c5f9-4f47-8f0b-5784a95159a4"}]: dispatch 2024-03-24T05:28:53.073 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:52 smithi063 ceph-mon[30595]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb781706-c5f9-4f47-8f0b-5784a95159a4"}]: dispatch 2024-03-24T05:28:53.073 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:52 smithi063 ceph-mon[30595]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb781706-c5f9-4f47-8f0b-5784a95159a4"}]': finished 2024-03-24T05:28:53.073 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:52 smithi063 ceph-mon[30595]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T05:28:53.073 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:52 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T05:28:53.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:52 smithi023 ceph-mon[25869]: from='client.? 172.21.15.63:0/2416552611' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb781706-c5f9-4f47-8f0b-5784a95159a4"}]: dispatch 2024-03-24T05:28:53.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:52 smithi023 ceph-mon[25869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb781706-c5f9-4f47-8f0b-5784a95159a4"}]: dispatch 2024-03-24T05:28:53.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:52 smithi023 ceph-mon[25869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb781706-c5f9-4f47-8f0b-5784a95159a4"}]': finished 2024-03-24T05:28:53.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:52 smithi023 ceph-mon[25869]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T05:28:53.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:52 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T05:28:54.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:54 smithi023 ceph-mon[25869]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:28:54.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:54 smithi023 ceph-mon[25869]: from='client.? 172.21.15.63:0/4062029596' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T05:28:54.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:54 smithi063 ceph-mon[30595]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:28:54.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:54 smithi063 ceph-mon[30595]: from='client.? 172.21.15.63:0/4062029596' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T05:28:54.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:54 smithi143 ceph-mon[30288]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:28:54.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:54 smithi143 ceph-mon[30288]: from='client.? 172.21.15.63:0/4062029596' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T05:28:56.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:56 smithi063 ceph-mon[30595]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:28:56.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:56 smithi023 ceph-mon[25869]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:28:56.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:56 smithi143 ceph-mon[30288]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:28:58.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:58 smithi063 ceph-mon[30595]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:28:58.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:58 smithi023 ceph-mon[25869]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:28:58.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:58 smithi143 ceph-mon[30288]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:28:59.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:59 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T05:28:59.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:28:59 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:59.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:59 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T05:28:59.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:28:59 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:28:59.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:59 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T05:28:59.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:28:59 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:00.478 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:00 smithi063 ceph-mon[30595]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:29:00.478 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:00 smithi063 ceph-mon[30595]: Deploying daemon osd.1 on smithi063 2024-03-24T05:29:00.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:00 smithi143 ceph-mon[30288]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:29:00.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:00 smithi143 ceph-mon[30288]: Deploying daemon osd.1 on smithi063 2024-03-24T05:29:00.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:00 smithi023 ceph-mon[25869]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:29:00.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:00 smithi023 ceph-mon[25869]: Deploying daemon osd.1 on smithi063 2024-03-24T05:29:02.240 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:02 smithi063 ceph-mon[30595]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:29:02.381 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:02 smithi023 ceph-mon[25869]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:29:02.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:02 smithi143 ceph-mon[30288]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:29:04.361 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:04 smithi063 ceph-mon[30595]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:29:04.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:04 smithi143 ceph-mon[30288]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:29:04.568 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:04 smithi023 ceph-mon[25869]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:29:05.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:05 smithi063 ceph-mon[30595]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:29:05.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:05 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:05.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:05 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:29:05.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:05 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:05.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:05 smithi143 ceph-mon[30288]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:29:05.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:05 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:05.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:05 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:29:05.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:05 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:05.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:05 smithi023 ceph-mon[25869]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:29:05.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:05 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:05.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:05 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:29:05.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:05 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:07.984 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 1 on host 'smithi063' 2024-03-24T05:29:08.278 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:07 smithi063 ceph-mon[30595]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:29:08.278 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:07 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:08.278 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:07 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:08.278 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:07 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:08.278 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:07 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:29:08.279 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:07 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:08.279 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:07 smithi063 ceph-mon[30595]: from='osd.1 [v2:172.21.15.63:6800/1403713781,v1:172.21.15.63:6801/1403713781]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T05:29:08.279 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:07 smithi063 ceph-mon[30595]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T05:29:08.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:07 smithi143 ceph-mon[30288]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:29:08.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:07 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:08.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:07 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:08.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:07 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:08.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:07 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:29:08.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:07 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:08.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:07 smithi143 ceph-mon[30288]: from='osd.1 [v2:172.21.15.63:6800/1403713781,v1:172.21.15.63:6801/1403713781]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T05:29:08.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:07 smithi143 ceph-mon[30288]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T05:29:08.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:07 smithi023 ceph-mon[25869]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:29:08.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:07 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:08.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:07 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:08.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:07 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:08.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:07 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:29:08.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:07 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:08.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:07 smithi023 ceph-mon[25869]: from='osd.1 [v2:172.21.15.63:6800/1403713781,v1:172.21.15.63:6801/1403713781]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T05:29:08.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:07 smithi023 ceph-mon[25869]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T05:29:08.604 DEBUG:teuthology.orchestra.run.smithi063:osd.1> sudo journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@osd.1.service 2024-03-24T05:29:08.607 INFO:tasks.cephadm:Deploying osd.2 on smithi143 with /dev/nvme4n1... 2024-03-24T05:29:08.607 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-24T05:29:08.850 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T05:29:09.148 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:08 smithi063 ceph-mon[30595]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T05:29:09.148 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:08 smithi063 ceph-mon[30595]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T05:29:09.148 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:08 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T05:29:09.148 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:08 smithi063 ceph-mon[30595]: from='osd.1 [v2:172.21.15.63:6800/1403713781,v1:172.21.15.63:6801/1403713781]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-24T05:29:09.148 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:08 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:09.148 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:08 smithi063 ceph-mon[30595]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-24T05:29:09.148 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:08 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:09.148 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:08 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:29:09.149 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:29:08 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-1[35167]: 2024-03-24T05:29:08.976+0000 7f93c6ef8700 -1 osd.1 0 waiting for initial osdmap 2024-03-24T05:29:09.149 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:29:08 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-1[35167]: 2024-03-24T05:29:08.978+0000 7f93c2ceb700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T05:29:09.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:08 smithi143 ceph-mon[30288]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T05:29:09.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:08 smithi143 ceph-mon[30288]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T05:29:09.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:08 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T05:29:09.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:08 smithi143 ceph-mon[30288]: from='osd.1 [v2:172.21.15.63:6800/1403713781,v1:172.21.15.63:6801/1403713781]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-24T05:29:09.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:08 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:09.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:08 smithi143 ceph-mon[30288]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-24T05:29:09.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:08 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:09.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:08 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:29:09.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:08 smithi023 ceph-mon[25869]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T05:29:09.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:08 smithi023 ceph-mon[25869]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T05:29:09.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:08 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T05:29:09.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:08 smithi023 ceph-mon[25869]: from='osd.1 [v2:172.21.15.63:6800/1403713781,v1:172.21.15.63:6801/1403713781]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-24T05:29:09.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:08 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:09.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:08 smithi023 ceph-mon[25869]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-24T05:29:09.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:08 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:09.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:08 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:29:10.152 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:09 smithi143 ceph-mon[30288]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:29:10.152 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:09 smithi143 ceph-mon[30288]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-24T05:29:10.152 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:09 smithi143 ceph-mon[30288]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T05:29:10.152 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:09 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T05:29:10.152 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:09 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T05:29:10.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:09 smithi063 ceph-mon[30595]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:29:10.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:09 smithi063 ceph-mon[30595]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-24T05:29:10.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:09 smithi063 ceph-mon[30595]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T05:29:10.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:09 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T05:29:10.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:09 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T05:29:10.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:09 smithi023 ceph-mon[25869]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:29:10.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:09 smithi023 ceph-mon[25869]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-24T05:29:10.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:09 smithi023 ceph-mon[25869]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T05:29:10.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:09 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T05:29:10.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:09 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T05:29:10.698 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping: /dev/nvme4n1 2024-03-24T05:29:10.698 INFO:teuthology.orchestra.run.smithi143.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-24T05:29:10.698 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-24T05:29:10.698 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10+0 records in 2024-03-24T05:29:10.698 INFO:teuthology.orchestra.run.smithi143.stderr:10+0 records out 2024-03-24T05:29:10.698 INFO:teuthology.orchestra.run.smithi143.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128035 s, 819 MB/s 2024-03-24T05:29:10.698 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping successful for: 2024-03-24T05:29:11.248 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:10 smithi143 ceph-mon[30288]: purged_snaps scrub starts 2024-03-24T05:29:11.248 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:10 smithi143 ceph-mon[30288]: purged_snaps scrub ok 2024-03-24T05:29:11.248 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:10 smithi143 ceph-mon[30288]: osd.1 [v2:172.21.15.63:6800/1403713781,v1:172.21.15.63:6801/1403713781] boot 2024-03-24T05:29:11.248 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:10 smithi143 ceph-mon[30288]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T05:29:11.248 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:10 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T05:29:11.274 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi143:/dev/nvme4n1 2024-03-24T05:29:11.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:10 smithi063 ceph-mon[30595]: purged_snaps scrub starts 2024-03-24T05:29:11.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:10 smithi063 ceph-mon[30595]: purged_snaps scrub ok 2024-03-24T05:29:11.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:10 smithi063 ceph-mon[30595]: osd.1 [v2:172.21.15.63:6800/1403713781,v1:172.21.15.63:6801/1403713781] boot 2024-03-24T05:29:11.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:10 smithi063 ceph-mon[30595]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T05:29:11.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:10 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T05:29:11.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:10 smithi023 ceph-mon[25869]: purged_snaps scrub starts 2024-03-24T05:29:11.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:10 smithi023 ceph-mon[25869]: purged_snaps scrub ok 2024-03-24T05:29:11.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:10 smithi023 ceph-mon[25869]: osd.1 [v2:172.21.15.63:6800/1403713781,v1:172.21.15.63:6801/1403713781] boot 2024-03-24T05:29:11.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:10 smithi023 ceph-mon[25869]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T05:29:11.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:10 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T05:29:11.506 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T05:29:12.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:11 smithi063 ceph-mon[30595]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:29:12.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:11 smithi063 ceph-mon[30595]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T05:29:12.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:11 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:12.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:11 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:12.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:11 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T05:29:12.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:11 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:12.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:11 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:12.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:11 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:29:12.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:11 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:12.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:11 smithi143 ceph-mon[30288]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:29:12.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:11 smithi143 ceph-mon[30288]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T05:29:12.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:11 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:12.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:11 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:12.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:11 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T05:29:12.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:11 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:12.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:11 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:12.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:11 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:29:12.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:11 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:12.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:11 smithi023 ceph-mon[25869]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T05:29:12.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:11 smithi023 ceph-mon[25869]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T05:29:12.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:11 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:12.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:11 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:12.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:11 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T05:29:12.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:11 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:12.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:11 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:12.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:11 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:29:12.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:11 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:13.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:12 smithi143 ceph-mon[30288]: Detected new or changed devices on smithi063 2024-03-24T05:29:13.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:12 smithi143 ceph-mon[30288]: Adjusting osd_memory_target on smithi063 to 17088M 2024-03-24T05:29:13.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:12 smithi063 ceph-mon[30595]: Detected new or changed devices on smithi063 2024-03-24T05:29:13.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:12 smithi063 ceph-mon[30595]: Adjusting osd_memory_target on smithi063 to 17088M 2024-03-24T05:29:13.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:12 smithi023 ceph-mon[25869]: Detected new or changed devices on smithi063 2024-03-24T05:29:13.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:12 smithi023 ceph-mon[25869]: Adjusting osd_memory_target on smithi063 to 17088M 2024-03-24T05:29:14.268 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:13 smithi143 ceph-mon[30288]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:14.269 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:13 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T05:29:14.269 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:13 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T05:29:14.269 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:13 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:14.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:13 smithi063 ceph-mon[30595]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:14.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:13 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T05:29:14.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:13 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T05:29:14.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:13 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:14.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:13 smithi023 ceph-mon[25869]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:14.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:13 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T05:29:14.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:13 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T05:29:14.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:13 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:15.182 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:15 smithi143 ceph-mon[30288]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:29:15.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:15 smithi063 ceph-mon[30595]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:29:15.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:15 smithi023 ceph-mon[25869]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:29:16.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:16 smithi063 ceph-mon[30595]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:16.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:16 smithi143 ceph-mon[30288]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:16.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:16 smithi023 ceph-mon[25869]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:17.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:17 smithi063 ceph-mon[30595]: from='client.? 172.21.15.143:0/1497715614' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "374d416e-cb02-420f-995d-0e6e035dd239"}]: dispatch 2024-03-24T05:29:17.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:17 smithi063 ceph-mon[30595]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "374d416e-cb02-420f-995d-0e6e035dd239"}]: dispatch 2024-03-24T05:29:17.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:17 smithi063 ceph-mon[30595]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "374d416e-cb02-420f-995d-0e6e035dd239"}]': finished 2024-03-24T05:29:17.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:17 smithi063 ceph-mon[30595]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T05:29:17.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:17 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T05:29:17.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:17 smithi143 ceph-mon[30288]: from='client.? 172.21.15.143:0/1497715614' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "374d416e-cb02-420f-995d-0e6e035dd239"}]: dispatch 2024-03-24T05:29:17.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:17 smithi143 ceph-mon[30288]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "374d416e-cb02-420f-995d-0e6e035dd239"}]: dispatch 2024-03-24T05:29:17.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:17 smithi143 ceph-mon[30288]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "374d416e-cb02-420f-995d-0e6e035dd239"}]': finished 2024-03-24T05:29:17.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:17 smithi143 ceph-mon[30288]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T05:29:17.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:17 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T05:29:17.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:17 smithi023 ceph-mon[25869]: from='client.? 172.21.15.143:0/1497715614' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "374d416e-cb02-420f-995d-0e6e035dd239"}]: dispatch 2024-03-24T05:29:17.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:17 smithi023 ceph-mon[25869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "374d416e-cb02-420f-995d-0e6e035dd239"}]: dispatch 2024-03-24T05:29:17.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:17 smithi023 ceph-mon[25869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "374d416e-cb02-420f-995d-0e6e035dd239"}]': finished 2024-03-24T05:29:17.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:17 smithi023 ceph-mon[25869]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T05:29:17.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:17 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T05:29:18.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:18 smithi063 ceph-mon[30595]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:18.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:18 smithi063 ceph-mon[30595]: from='client.? 172.21.15.143:0/201052361' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T05:29:18.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:18 smithi143 ceph-mon[30288]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:18.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:18 smithi143 ceph-mon[30288]: from='client.? 172.21.15.143:0/201052361' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T05:29:18.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:18 smithi023 ceph-mon[25869]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:18.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:18 smithi023 ceph-mon[25869]: from='client.? 172.21.15.143:0/201052361' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T05:29:20.059 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:20 smithi143 ceph-mon[30288]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:20.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:20 smithi063 ceph-mon[30595]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:20.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:20 smithi023 ceph-mon[25869]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:22.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:22 smithi063 ceph-mon[30595]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:22.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:22 smithi143 ceph-mon[30288]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:22.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:22 smithi023 ceph-mon[25869]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:23.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:23 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T05:29:23.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:23 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:23.309 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:23 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T05:29:23.309 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:23 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:23.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:23 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T05:29:23.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:23 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:24.302 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:24 smithi143 ceph-mon[30288]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:24.302 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:24 smithi143 ceph-mon[30288]: Deploying daemon osd.2 on smithi143 2024-03-24T05:29:24.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:24 smithi023 ceph-mon[25869]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:24.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:24 smithi023 ceph-mon[25869]: Deploying daemon osd.2 on smithi143 2024-03-24T05:29:24.553 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:24 smithi063 ceph-mon[30595]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:24.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:24 smithi063 ceph-mon[30595]: Deploying daemon osd.2 on smithi143 2024-03-24T05:29:26.261 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:26 smithi143 ceph-mon[30288]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:26.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:26 smithi023 ceph-mon[25869]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:26.553 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:26 smithi063 ceph-mon[30595]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:28.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:28 smithi023 ceph-mon[25869]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:28.412 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:28 smithi143 ceph-mon[30288]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:28.553 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:28 smithi063 ceph-mon[30595]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:30.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:29 smithi063 ceph-mon[30595]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:30.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:29 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:30.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:29 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:30.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:29 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:29:30.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:29 smithi143 ceph-mon[30288]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:30.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:29 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:30.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:29 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:30.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:29 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:29:30.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:29 smithi023 ceph-mon[25869]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:30.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:29 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:30.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:29 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:30.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:29 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:29:31.985 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:31 smithi143 ceph-mon[30288]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:31.986 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:31 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:31.986 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:31 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:31.986 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:31 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:31.986 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:31 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:29:31.986 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:31 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:32.053 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:31 smithi063 ceph-mon[30595]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:32.053 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:31 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:32.053 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:31 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:32.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:31 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:32.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:31 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:29:32.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:31 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:32.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:31 smithi023 ceph-mon[25869]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:32.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:31 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:32.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:31 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:32.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:31 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:32.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:31 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:29:32.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:31 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:32.362 INFO:teuthology.orchestra.run.smithi143.stdout:Created osd(s) 2 on host 'smithi143' 2024-03-24T05:29:32.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:32 smithi143 ceph-mon[30288]: from='osd.2 [v2:172.21.15.143:6800/573156693,v1:172.21.15.143:6801/573156693]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T05:29:32.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:32 smithi143 ceph-mon[30288]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T05:29:32.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:32 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:32.959 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:32 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:32.959 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:32 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:29:32.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:32 smithi023 ceph-mon[25869]: from='osd.2 [v2:172.21.15.143:6800/573156693,v1:172.21.15.143:6801/573156693]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T05:29:32.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:32 smithi023 ceph-mon[25869]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T05:29:32.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:32 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:32.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:32 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:32.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:32 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:29:32.984 DEBUG:teuthology.orchestra.run.smithi143:osd.2> sudo journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@osd.2.service 2024-03-24T05:29:32.987 INFO:tasks.cephadm:Waiting for 3 OSDs to come up... 2024-03-24T05:29:32.988 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T05:29:33.053 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:32 smithi063 ceph-mon[30595]: from='osd.2 [v2:172.21.15.143:6800/573156693,v1:172.21.15.143:6801/573156693]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T05:29:33.053 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:32 smithi063 ceph-mon[30595]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T05:29:33.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:32 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:33.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:32 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:33.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:32 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:29:33.238 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:29:33.986 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:33 smithi023 ceph-mon[25869]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:33.986 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:33 smithi023 ceph-mon[25869]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T05:29:33.986 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:33 smithi023 ceph-mon[25869]: from='osd.2 [v2:172.21.15.143:6800/573156693,v1:172.21.15.143:6801/573156693]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T05:29:33.986 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:33 smithi023 ceph-mon[25869]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T05:29:33.986 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:33 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T05:29:33.986 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:33 smithi023 ceph-mon[25869]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T05:29:34.012 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:33 smithi143 ceph-mon[30288]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:34.012 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:33 smithi143 ceph-mon[30288]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T05:29:34.012 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:33 smithi143 ceph-mon[30288]: from='osd.2 [v2:172.21.15.143:6800/573156693,v1:172.21.15.143:6801/573156693]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T05:29:34.012 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:33 smithi143 ceph-mon[30288]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T05:29:34.012 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:33 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T05:29:34.012 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:33 smithi143 ceph-mon[30288]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T05:29:34.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:33 smithi063 ceph-mon[30595]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:34.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:33 smithi063 ceph-mon[30595]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T05:29:34.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:33 smithi063 ceph-mon[30595]: from='osd.2 [v2:172.21.15.143:6800/573156693,v1:172.21.15.143:6801/573156693]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T05:29:34.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:33 smithi063 ceph-mon[30595]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T05:29:34.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:33 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T05:29:34.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:33 smithi063 ceph-mon[30595]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T05:29:34.283 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:29:34 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-2[35531]: 2024-03-24T05:29:34.126+0000 7ff89d6ed700 -1 osd.2 0 waiting for initial osdmap 2024-03-24T05:29:34.283 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:29:34 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-2[35531]: 2024-03-24T05:29:34.128+0000 7ff894cd7700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T05:29:34.993 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:29:34.994 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:34 smithi023 ceph-mon[25869]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T05:29:34.994 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:34 smithi023 ceph-mon[25869]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T05:29:34.994 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:34 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T05:29:34.994 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:34 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T05:29:34.994 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:34 smithi023 ceph-mon[25869]: from='osd.2 ' entity='osd.2' 2024-03-24T05:29:35.053 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:34 smithi063 ceph-mon[30595]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T05:29:35.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:34 smithi063 ceph-mon[30595]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T05:29:35.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:34 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T05:29:35.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:34 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T05:29:35.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:34 smithi063 ceph-mon[30595]: from='osd.2 ' entity='osd.2' 2024-03-24T05:29:35.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:34 smithi143 ceph-mon[30288]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T05:29:35.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:34 smithi143 ceph-mon[30288]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T05:29:35.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:34 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T05:29:35.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:34 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T05:29:35.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:34 smithi143 ceph-mon[30288]: from='osd.2 ' entity='osd.2' 2024-03-24T05:29:35.502 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":17,"num_osds":3,"num_up_osds":2,"osd_up_since":1711258149,"num_in_osds":3,"osd_in_since":1711258156,"num_remapped_pgs":0} 2024-03-24T05:29:35.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:35 smithi023 ceph-mon[25869]: purged_snaps scrub starts 2024-03-24T05:29:35.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:35 smithi023 ceph-mon[25869]: purged_snaps scrub ok 2024-03-24T05:29:35.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:35 smithi023 ceph-mon[25869]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:35.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:35 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T05:29:35.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:35 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/4195866570' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T05:29:35.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:35 smithi023 ceph-mon[25869]: osd.2 [v2:172.21.15.143:6800/573156693,v1:172.21.15.143:6801/573156693] boot 2024-03-24T05:29:35.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:35 smithi023 ceph-mon[25869]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T05:29:35.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:35 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T05:29:35.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:35 smithi143 ceph-mon[30288]: purged_snaps scrub starts 2024-03-24T05:29:35.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:35 smithi143 ceph-mon[30288]: purged_snaps scrub ok 2024-03-24T05:29:35.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:35 smithi143 ceph-mon[30288]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:35.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:35 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T05:29:35.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:35 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/4195866570' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T05:29:35.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:35 smithi143 ceph-mon[30288]: osd.2 [v2:172.21.15.143:6800/573156693,v1:172.21.15.143:6801/573156693] boot 2024-03-24T05:29:35.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:35 smithi143 ceph-mon[30288]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T05:29:35.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:35 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T05:29:36.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:35 smithi063 ceph-mon[30595]: purged_snaps scrub starts 2024-03-24T05:29:36.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:35 smithi063 ceph-mon[30595]: purged_snaps scrub ok 2024-03-24T05:29:36.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:35 smithi063 ceph-mon[30595]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T05:29:36.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:35 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T05:29:36.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:35 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/4195866570' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T05:29:36.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:35 smithi063 ceph-mon[30595]: osd.2 [v2:172.21.15.143:6800/573156693,v1:172.21.15.143:6801/573156693] boot 2024-03-24T05:29:36.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:35 smithi063 ceph-mon[30595]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T05:29:36.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:35 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T05:29:36.504 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T05:29:36.740 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:29:37.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:37 smithi023 ceph-mon[25869]: Detected new or changed devices on smithi143 2024-03-24T05:29:37.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:37 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:37.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:37 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:37.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:37 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T05:29:37.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:37 smithi023 ceph-mon[25869]: Adjusting osd_memory_target on smithi143 to 17088M 2024-03-24T05:29:37.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:37 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:37.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:37 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:37.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:37 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:29:37.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:37 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:37.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:37 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T05:29:37.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:37 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T05:29:37.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:37 smithi023 ceph-mon[25869]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T05:29:37.104 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:37 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T05:29:37.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:37 smithi063 ceph-mon[30595]: Detected new or changed devices on smithi143 2024-03-24T05:29:37.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:37 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:37.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:37 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:37.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:37 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T05:29:37.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:37 smithi063 ceph-mon[30595]: Adjusting osd_memory_target on smithi143 to 17088M 2024-03-24T05:29:37.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:37 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:37.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:37 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:37.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:37 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:29:37.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:37 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:37.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:37 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T05:29:37.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:37 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T05:29:37.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:37 smithi063 ceph-mon[30595]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T05:29:37.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:37 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T05:29:37.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:37 smithi143 ceph-mon[30288]: Detected new or changed devices on smithi143 2024-03-24T05:29:37.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:37 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:37.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:37 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:37.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:37 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T05:29:37.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:37 smithi143 ceph-mon[30288]: Adjusting osd_memory_target on smithi143 to 17088M 2024-03-24T05:29:37.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:37 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:37.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:37 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:37.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:37 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:29:37.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:37 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:37.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:37 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T05:29:37.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:37 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T05:29:37.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:37 smithi143 ceph-mon[30288]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T05:29:37.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:37 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T05:29:38.193 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:38 smithi023 ceph-mon[25869]: pgmap v81: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:29:38.193 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:38 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T05:29:38.194 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:38 smithi023 ceph-mon[25869]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T05:29:38.194 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:29:38 smithi023 sudo[46871]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-24T05:29:38.194 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:29:38 smithi023 sudo[46871]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T05:29:38.194 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:29:38 smithi023 sudo[46871]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T05:29:38.194 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:29:38 smithi023 sudo[46871]: pam_unix(sudo:session): session closed for user root 2024-03-24T05:29:38.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:38 smithi063 ceph-mon[30595]: pgmap v81: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:29:38.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:38 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T05:29:38.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:38 smithi063 ceph-mon[30595]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T05:29:38.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:38 smithi143 ceph-mon[30288]: pgmap v81: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:29:38.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:38 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T05:29:38.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:38 smithi143 ceph-mon[30288]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T05:29:38.579 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:29:38 smithi023 sudo[46893]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-24T05:29:38.580 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:29:38 smithi023 sudo[46893]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T05:29:38.580 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:29:38 smithi023 sudo[46893]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T05:29:38.580 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:29:38 smithi023 sudo[46893]: pam_unix(sudo:session): session closed for user root 2024-03-24T05:29:38.626 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:29:38.850 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:29:38 smithi063 sudo[40814]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-24T05:29:38.850 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:29:38 smithi063 sudo[40814]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T05:29:38.850 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:29:38 smithi063 sudo[40814]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T05:29:38.850 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:29:38 smithi063 sudo[40814]: pam_unix(sudo:session): session closed for user root 2024-03-24T05:29:39.274 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":20,"num_osds":3,"num_up_osds":3,"osd_up_since":1711258175,"num_in_osds":3,"osd_in_since":1711258156,"num_remapped_pgs":0} 2024-03-24T05:29:39.277 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T05:29:39.304 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:29:38 smithi063 sudo[40817]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-24T05:29:39.304 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:29:38 smithi063 sudo[40817]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T05:29:39.304 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:29:38 smithi063 sudo[40817]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T05:29:39.304 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:29:38 smithi063 sudo[40817]: pam_unix(sudo:session): session closed for user root 2024-03-24T05:29:39.370 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:39 smithi143 ceph-mon[30288]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:29:39.370 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:39 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:39.370 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:39 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/2442633278' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T05:29:39.370 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:39 smithi143 ceph-mon[30288]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T05:29:39.370 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:39 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:39.370 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:29:39 smithi143 sudo[40632]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-24T05:29:39.371 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:29:39 smithi143 sudo[40632]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T05:29:39.371 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:29:39 smithi143 sudo[40632]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T05:29:39.371 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:29:39 smithi143 sudo[40632]: pam_unix(sudo:session): session closed for user root 2024-03-24T05:29:39.518 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:29:39.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:39 smithi023 ceph-mon[25869]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:29:39.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:39 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:39.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:39 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/2442633278' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T05:29:39.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:39 smithi023 ceph-mon[25869]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T05:29:39.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:39 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:39.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:39 smithi063 ceph-mon[30595]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:29:39.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:39 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:39.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:39 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/2442633278' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T05:29:39.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:39 smithi063 ceph-mon[30595]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T05:29:39.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:39 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:39.807 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:29:39 smithi143 sudo[40635]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-24T05:29:39.808 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:29:39 smithi143 sudo[40635]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T05:29:39.808 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:29:39 smithi143 sudo[40635]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T05:29:39.808 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:29:39 smithi143 sudo[40635]: pam_unix(sudo:session): session closed for user root 2024-03-24T05:29:39.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:39 smithi023 sudo[47017]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T05:29:39.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:39 smithi023 sudo[47017]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T05:29:39.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:39 smithi023 sudo[47017]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T05:29:39.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:39 smithi023 sudo[47017]: pam_unix(sudo:session): session closed for user root 2024-03-24T05:29:40.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:39 smithi023 sudo[47039]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T05:29:40.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:39 smithi023 sudo[47039]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T05:29:40.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:39 smithi023 sudo[47039]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T05:29:40.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:39 smithi023 sudo[47039]: pam_unix(sudo:session): session closed for user root 2024-03-24T05:29:40.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:40 smithi143 sudo[40638]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T05:29:40.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:40 smithi143 sudo[40638]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T05:29:40.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:40 smithi143 sudo[40638]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T05:29:40.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:40 smithi143 sudo[40638]: pam_unix(sudo:session): session closed for user root 2024-03-24T05:29:40.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:40 smithi143 sudo[40641]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T05:29:40.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:40 smithi143 sudo[40641]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T05:29:40.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:40 smithi143 sudo[40641]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T05:29:40.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:40 smithi143 sudo[40641]: pam_unix(sudo:session): session closed for user root 2024-03-24T05:29:40.715 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:40 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:40.715 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:40 smithi063 ceph-mon[30595]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T05:29:40.715 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:40 smithi063 ceph-mon[30595]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T05:29:40.716 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:40 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T05:29:40.716 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:40 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:29:40.716 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:40 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:29:40.716 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:40 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:40.716 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:40 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T05:29:40.716 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:40 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:29:40.716 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:40 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:29:40.737 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:40 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:40.737 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:40 smithi023 ceph-mon[25869]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T05:29:40.737 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:40 smithi023 ceph-mon[25869]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T05:29:40.737 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:40 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T05:29:40.738 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:40 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:29:40.738 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:40 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:29:40.738 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:40 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:40.738 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:40 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T05:29:40.738 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:40 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:29:40.738 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:40 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:29:40.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:40 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:40.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:40 smithi143 ceph-mon[30288]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T05:29:40.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:40 smithi143 ceph-mon[30288]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T05:29:40.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:40 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T05:29:40.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:40 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:29:40.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:40 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:29:40.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:40 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:40.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:40 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T05:29:40.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:40 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:29:40.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:40 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:29:40.979 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:40 smithi063 sudo[40820]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T05:29:40.979 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:40 smithi063 sudo[40820]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T05:29:40.979 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:40 smithi063 sudo[40820]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T05:29:40.979 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:40 smithi063 sudo[40820]: pam_unix(sudo:session): session closed for user root 2024-03-24T05:29:41.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:40 smithi063 sudo[40823]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T05:29:41.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:40 smithi063 sudo[40823]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T05:29:41.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:40 smithi063 sudo[40823]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T05:29:41.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:40 smithi063 sudo[40823]: pam_unix(sudo:session): session closed for user root 2024-03-24T05:29:41.475 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:29:41.475 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":21,"fsid":"e2aa55c2-e99e-11ee-95cd-87774f69a715","created":"2024-03-24T05:26:10.742267+0000","modified":"2024-03-24T05:29:38.687092+0000","last_up_change":"2024-03-24T05:29:35.130293+0000","last_in_change":"2024-03-24T05:29:16.039751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T05:29:36.514453+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38483f5a-088a-4caa-9228-5502305de04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":489464929},{"type":"v1","addr":"172.21.15.23:6803","nonce":489464929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":489464929},{"type":"v1","addr":"172.21.15.23:6805","nonce":489464929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":489464929},{"type":"v1","addr":"172.21.15.23:6809","nonce":489464929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":489464929},{"type":"v1","addr":"172.21.15.23:6807","nonce":489464929}]},"public_addr":"172.21.15.23:6803/489464929","cluster_addr":"172.21.15.23:6805/489464929","heartbeat_back_addr":"172.21.15.23:6809/489464929","heartbeat_front_addr":"172.21.15.23:6807/489464929","state":["exists","up"]},{"osd":1,"uuid":"bb781706-c5f9-4f47-8f0b-5784a95159a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1403713781},{"type":"v1","addr":"172.21.15.63:6801","nonce":1403713781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1403713781},{"type":"v1","addr":"172.21.15.63:6803","nonce":1403713781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1403713781},{"type":"v1","addr":"172.21.15.63:6807","nonce":1403713781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1403713781},{"type":"v1","addr":"172.21.15.63:6805","nonce":1403713781}]},"public_addr":"172.21.15.63:6801/1403713781","cluster_addr":"172.21.15.63:6803/1403713781","heartbeat_back_addr":"172.21.15.63:6807/1403713781","heartbeat_front_addr":"172.21.15.63:6805/1403713781","state":["exists","up"]},{"osd":2,"uuid":"374d416e-cb02-420f-995d-0e6e035dd239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":573156693},{"type":"v1","addr":"172.21.15.143:6801","nonce":573156693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":573156693},{"type":"v1","addr":"172.21.15.143:6803","nonce":573156693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":573156693},{"type":"v1","addr":"172.21.15.143:6807","nonce":573156693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":573156693},{"type":"v1","addr":"172.21.15.143:6805","nonce":573156693}]},"public_addr":"172.21.15.143:6801/573156693","cluster_addr":"172.21.15.143:6803/573156693","heartbeat_back_addr":"172.21.15.143:6807/573156693","heartbeat_front_addr":"172.21.15.143:6805/573156693","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T05:28:44.538609+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T05:29:08.577419+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T05:29:32.791088+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/477877701":"2024-03-25T05:27:10.172457+0000","172.21.15.23:0/3117010616":"2024-03-25T05:27:10.172457+0000","172.21.15.23:0/2279719634":"2024-03-25T05:26:43.320335+0000","172.21.15.23:6800/3490012969":"2024-03-25T05:26:43.320335+0000","172.21.15.23:6800/477877701":"2024-03-25T05:27:10.172457+0000","172.21.15.23:0/828298596":"2024-03-25T05:27:10.172457+0000","172.21.15.23:0/1184189117":"2024-03-25T05:27:10.172457+0000","172.21.15.23:0/1787617234":"2024-03-25T05:26:43.320335+0000","172.21.15.23:6801/3490012969":"2024-03-25T05:26:43.320335+0000","172.21.15.23:0/1789110770":"2024-03-25T05:26:43.320335+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T05:29:41.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:41 smithi023 ceph-mon[25869]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T05:29:41.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:41 smithi023 ceph-mon[25869]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:29:41.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:41 smithi023 ceph-mon[25869]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T05:29:41.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:41 smithi023 ceph-mon[25869]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-24T05:29:41.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:41 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:41.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:41 smithi023 ceph-mon[25869]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T05:29:41.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:41 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T05:29:41.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:41 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:29:41.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:41 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:29:41.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:41 smithi023 ceph-mon[25869]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T05:29:41.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:41 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:41.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:41 smithi063 ceph-mon[30595]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T05:29:41.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:41 smithi063 ceph-mon[30595]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:29:41.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:41 smithi063 ceph-mon[30595]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T05:29:41.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:41 smithi063 ceph-mon[30595]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-24T05:29:41.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:41 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:41.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:41 smithi063 ceph-mon[30595]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T05:29:41.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:41 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T05:29:41.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:41 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:29:41.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:41 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:29:41.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:41 smithi063 ceph-mon[30595]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T05:29:41.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:41 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:41.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:41 smithi143 ceph-mon[30288]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T05:29:41.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:41 smithi143 ceph-mon[30288]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:29:41.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:41 smithi143 ceph-mon[30288]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T05:29:41.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:41 smithi143 ceph-mon[30288]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-24T05:29:41.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:41 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:41.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:41 smithi143 ceph-mon[30288]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T05:29:41.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:41 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T05:29:41.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:41 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:29:41.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:41 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:29:41.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:41 smithi143 ceph-mon[30288]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T05:29:41.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:41 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:42.077 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T05:29:36.514453+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T05:29:42.078 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-24T05:29:42.314 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:29:42.688 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:42 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/3277691836' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T05:29:42.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:42 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/3277691836' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T05:29:42.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:42 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/3277691836' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T05:29:43.676 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:43 smithi023 ceph-mon[25869]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:29:43.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:43 smithi063 ceph-mon[30595]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:29:43.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:43 smithi143 ceph-mon[30288]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:29:43.988 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-24T05:29:44.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:44 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/3563839248' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T05:29:44.624 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi023 2024-03-24T05:29:44.624 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi023=foo.a' 2024-03-24T05:29:44.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:44 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/3563839248' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T05:29:44.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:44 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/3563839248' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T05:29:44.874 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T05:29:45.666 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:45 smithi143 ceph-mon[30288]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:29:45.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:45 smithi063 ceph-mon[30595]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:29:45.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:45 smithi023 ceph-mon[25869]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:29:46.664 INFO:teuthology.orchestra.run.smithi143.stdout:Scheduled rgw.foo.a update... 2024-03-24T05:29:47.224 DEBUG:teuthology.orchestra.run.smithi023:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@rgw.foo.a.service 2024-03-24T05:29:47.227 INFO:tasks.cephadm:Adding prometheus.a on smithi063 2024-03-24T05:29:47.227 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph orch apply prometheus '1;smithi063=a' 2024-03-24T05:29:47.459 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T05:29:47.901 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:47 smithi143 ceph-mon[30288]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-24T05:29:47.901 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:47 smithi143 ceph-mon[30288]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi023=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:29:47.901 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:47 smithi143 ceph-mon[30288]: Saving service rgw.foo.a spec with placement smithi023=foo.a;count:1 2024-03-24T05:29:47.901 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:47 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:47.901 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:47 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:29:47.901 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:47 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:47.901 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:47 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:29:47.902 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:47 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:47.902 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:47 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T05:29:47.902 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:47 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T05:29:47.902 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:47 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:47.902 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:47 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:47.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:47 smithi023 ceph-mon[25869]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-24T05:29:47.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:47 smithi023 ceph-mon[25869]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi023=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:29:47.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:47 smithi023 ceph-mon[25869]: Saving service rgw.foo.a spec with placement smithi023=foo.a;count:1 2024-03-24T05:29:47.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:47 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:47.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:47 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:29:47.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:47 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:47.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:47 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:29:47.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:47 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:47.933 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:47 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T05:29:47.933 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:47 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T05:29:47.933 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:47 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:47.933 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:47 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:48.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:47 smithi063 ceph-mon[30595]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-24T05:29:48.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:47 smithi063 ceph-mon[30595]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi023=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:29:48.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:47 smithi063 ceph-mon[30595]: Saving service rgw.foo.a spec with placement smithi023=foo.a;count:1 2024-03-24T05:29:48.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:47 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:48.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:47 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:29:48.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:47 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:48.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:47 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:29:48.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:47 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:48.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:47 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T05:29:48.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:47 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T05:29:48.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:47 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:48.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:47 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:48.799 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:48 smithi143 ceph-mon[30288]: Deploying daemon rgw.foo.a on smithi023 2024-03-24T05:29:48.802 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:48 smithi023 ceph-mon[25869]: Deploying daemon rgw.foo.a on smithi023 2024-03-24T05:29:49.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:48 smithi063 ceph-mon[30595]: Deploying daemon rgw.foo.a on smithi023 2024-03-24T05:29:49.220 INFO:teuthology.orchestra.run.smithi143.stdout:Scheduled prometheus update... 2024-03-24T05:29:49.472 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 24 05:29:49 smithi023 systemd[1]: Starting Ceph rgw.foo.a for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:29:49.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:49 smithi023 ceph-mon[25869]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-24T05:29:49.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:49 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:49.782 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 24 05:29:49 smithi023 podman[47671]: 2024-03-24 05:29:49.472852448 +0000 UTC m=+0.113725542 container create 29ebffacb9c38b8a5850a171f9641d1a000e1260dd8a7c227b37d699bd9c45df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-rgw-foo-a, RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-24T05:29:49.782 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 24 05:29:49 smithi023 podman[47671]: 2024-03-24 05:29:49.376493122 +0000 UTC m=+0.017366216 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T05:29:49.782 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 24 05:29:49 smithi023 podman[47671]: 2024-03-24 05:29:49.683569562 +0000 UTC m=+0.324442657 container init 29ebffacb9c38b8a5850a171f9641d1a000e1260dd8a7c227b37d699bd9c45df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-rgw-foo-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b) 2024-03-24T05:29:49.783 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 24 05:29:49 smithi023 podman[47671]: 2024-03-24 05:29:49.688957666 +0000 UTC m=+0.329830760 container start 29ebffacb9c38b8a5850a171f9641d1a000e1260dd8a7c227b37d699bd9c45df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-24T05:29:49.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:49 smithi143 ceph-mon[30288]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-24T05:29:49.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:49 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:49.844 DEBUG:teuthology.orchestra.run.smithi063:prometheus.a> sudo journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@prometheus.a.service 2024-03-24T05:29:49.848 INFO:tasks.cephadm:Adding node-exporter.a on smithi023 2024-03-24T05:29:49.848 INFO:tasks.cephadm:Adding node-exporter.b on smithi063 2024-03-24T05:29:49.848 INFO:tasks.cephadm:Adding node-exporter.c on smithi143 2024-03-24T05:29:49.849 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph orch apply node-exporter '3;smithi023=a;smithi063=b;smithi143=c' 2024-03-24T05:29:50.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:49 smithi063 ceph-mon[30595]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-24T05:29:50.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:49 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:50.079 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 24 05:29:49 smithi023 bash[47671]: 29ebffacb9c38b8a5850a171f9641d1a000e1260dd8a7c227b37d699bd9c45df 2024-03-24T05:29:50.079 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 24 05:29:49 smithi023 systemd[1]: Started Ceph rgw.foo.a for e2aa55c2-e99e-11ee-95cd-87774f69a715. 2024-03-24T05:29:50.083 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T05:29:51.176 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:50 smithi143 ceph-mon[30288]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi063=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:29:51.176 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:50 smithi143 ceph-mon[30288]: Saving service prometheus spec with placement smithi063=a;count:1 2024-03-24T05:29:51.176 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:50 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:51.176 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:50 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:51.176 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:50 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:51.176 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:50 smithi143 ceph-mon[30288]: Saving service rgw.foo.a spec with placement smithi023=foo.a;count:1 2024-03-24T05:29:51.176 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:50 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:51.176 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:50 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:51.176 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:50 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:29:51.176 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:50 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:51.176 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:50 smithi143 ceph-mon[30288]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T05:29:51.176 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:50 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/3140928910' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T05:29:51.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:50 smithi063 ceph-mon[30595]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi063=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:29:51.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:50 smithi063 ceph-mon[30595]: Saving service prometheus spec with placement smithi063=a;count:1 2024-03-24T05:29:51.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:50 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:51.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:50 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:51.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:50 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:51.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:50 smithi063 ceph-mon[30595]: Saving service rgw.foo.a spec with placement smithi023=foo.a;count:1 2024-03-24T05:29:51.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:50 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:51.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:50 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:51.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:50 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:29:51.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:50 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:51.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:50 smithi063 ceph-mon[30595]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T05:29:51.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:50 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/3140928910' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T05:29:51.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:50 smithi023 ceph-mon[25869]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi063=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:29:51.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:50 smithi023 ceph-mon[25869]: Saving service prometheus spec with placement smithi063=a;count:1 2024-03-24T05:29:51.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:50 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:51.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:50 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:51.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:50 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:51.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:50 smithi023 ceph-mon[25869]: Saving service rgw.foo.a spec with placement smithi023=foo.a;count:1 2024-03-24T05:29:51.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:50 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:51.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:50 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:51.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:50 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:29:51.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:50 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:51.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:50 smithi023 ceph-mon[25869]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T05:29:51.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:50 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/3140928910' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T05:29:51.995 INFO:teuthology.orchestra.run.smithi143.stdout:Scheduled node-exporter update... 2024-03-24T05:29:52.499 DEBUG:teuthology.orchestra.run.smithi023:node-exporter.a> sudo journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@node-exporter.a.service 2024-03-24T05:29:52.501 DEBUG:teuthology.orchestra.run.smithi063:node-exporter.b> sudo journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@node-exporter.b.service 2024-03-24T05:29:52.503 DEBUG:teuthology.orchestra.run.smithi143:node-exporter.c> sudo journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@node-exporter.c.service 2024-03-24T05:29:52.505 INFO:tasks.cephadm:Adding alertmanager.a on smithi023 2024-03-24T05:29:52.506 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph orch apply alertmanager '1;smithi023=a' 2024-03-24T05:29:52.553 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:52 smithi063 ceph-mon[30595]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-24T05:29:52.553 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:52 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:52.553 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:52 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:52.553 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:52 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:52.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:52 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:29:52.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:52 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/3140928910' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T05:29:52.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:52 smithi063 ceph-mon[30595]: osdmap e23: 3 total, 3 up, 3 in 2024-03-24T05:29:52.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:52 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:52.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:52 smithi143 ceph-mon[30288]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-24T05:29:52.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:52 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:52.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:52 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:52.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:52 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:52.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:52 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:29:52.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:52 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/3140928910' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T05:29:52.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:52 smithi143 ceph-mon[30288]: osdmap e23: 3 total, 3 up, 3 in 2024-03-24T05:29:52.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:52 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:52.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:52 smithi023 ceph-mon[25869]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-24T05:29:52.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:52 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:52.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:52 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:52.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:52 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:29:52.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:52 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:29:52.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:52 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/3140928910' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T05:29:52.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:52 smithi023 ceph-mon[25869]: osdmap e23: 3 total, 3 up, 3 in 2024-03-24T05:29:52.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:52 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:52.784 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T05:29:53.553 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:53 smithi063 ceph-mon[30595]: Checking dashboard <-> RGW credentials 2024-03-24T05:29:53.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:53 smithi063 ceph-mon[30595]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi023=a;smithi063=b;smithi143=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:29:53.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:53 smithi063 ceph-mon[30595]: Saving service node-exporter spec with placement smithi023=a;smithi063=b;smithi143=c;count:3 2024-03-24T05:29:53.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:53 smithi063 ceph-mon[30595]: pgmap v94: 33 pgs: 24 creating+peering, 8 unknown, 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-24T05:29:53.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:53 smithi063 ceph-mon[30595]: osdmap e24: 3 total, 3 up, 3 in 2024-03-24T05:29:53.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:53 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T05:29:53.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:53 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T05:29:53.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:53 smithi143 ceph-mon[30288]: Checking dashboard <-> RGW credentials 2024-03-24T05:29:53.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:53 smithi143 ceph-mon[30288]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi023=a;smithi063=b;smithi143=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:29:53.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:53 smithi143 ceph-mon[30288]: Saving service node-exporter spec with placement smithi023=a;smithi063=b;smithi143=c;count:3 2024-03-24T05:29:53.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:53 smithi143 ceph-mon[30288]: pgmap v94: 33 pgs: 24 creating+peering, 8 unknown, 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-24T05:29:53.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:53 smithi143 ceph-mon[30288]: osdmap e24: 3 total, 3 up, 3 in 2024-03-24T05:29:53.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:53 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T05:29:53.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:53 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T05:29:53.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:53 smithi023 ceph-mon[25869]: Checking dashboard <-> RGW credentials 2024-03-24T05:29:53.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:53 smithi023 ceph-mon[25869]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi023=a;smithi063=b;smithi143=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:29:53.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:53 smithi023 ceph-mon[25869]: Saving service node-exporter spec with placement smithi023=a;smithi063=b;smithi143=c;count:3 2024-03-24T05:29:53.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:53 smithi023 ceph-mon[25869]: pgmap v94: 33 pgs: 24 creating+peering, 8 unknown, 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-24T05:29:53.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:53 smithi023 ceph-mon[25869]: osdmap e24: 3 total, 3 up, 3 in 2024-03-24T05:29:53.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:53 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T05:29:53.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:53 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T05:29:54.625 INFO:teuthology.orchestra.run.smithi143.stdout:Scheduled alertmanager update... 2024-03-24T05:29:55.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:54 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T05:29:55.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:54 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T05:29:55.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:54 smithi063 ceph-mon[30595]: osdmap e25: 3 total, 3 up, 3 in 2024-03-24T05:29:55.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:54 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:55.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:54 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T05:29:55.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:54 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T05:29:55.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:54 smithi143 ceph-mon[30288]: osdmap e25: 3 total, 3 up, 3 in 2024-03-24T05:29:55.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:54 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:55.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:54 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T05:29:55.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:54 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T05:29:55.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:54 smithi023 ceph-mon[25869]: osdmap e25: 3 total, 3 up, 3 in 2024-03-24T05:29:55.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:54 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:55.237 DEBUG:teuthology.orchestra.run.smithi023:alertmanager.a> sudo journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@alertmanager.a.service 2024-03-24T05:29:55.240 INFO:tasks.cephadm:Adding grafana.a on smithi063 2024-03-24T05:29:55.240 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph orch apply grafana '1;smithi063=a' 2024-03-24T05:29:55.509 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T05:29:55.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:55 smithi143 ceph-mon[30288]: pgmap v97: 65 pgs: 32 unknown, 24 creating+peering, 9 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 511 B/s wr, 0 op/s 2024-03-24T05:29:55.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:55 smithi143 ceph-mon[30288]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi023=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:29:55.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:55 smithi143 ceph-mon[30288]: Saving service alertmanager spec with placement smithi023=a;count:1 2024-03-24T05:29:55.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:55 smithi143 ceph-mon[30288]: osdmap e26: 3 total, 3 up, 3 in 2024-03-24T05:29:55.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:55 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T05:29:55.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:55 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T05:29:56.053 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:55 smithi063 ceph-mon[30595]: pgmap v97: 65 pgs: 32 unknown, 24 creating+peering, 9 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 511 B/s wr, 0 op/s 2024-03-24T05:29:56.053 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:55 smithi063 ceph-mon[30595]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi023=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:29:56.053 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:55 smithi063 ceph-mon[30595]: Saving service alertmanager spec with placement smithi023=a;count:1 2024-03-24T05:29:56.053 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:55 smithi063 ceph-mon[30595]: osdmap e26: 3 total, 3 up, 3 in 2024-03-24T05:29:56.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:55 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T05:29:56.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:55 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T05:29:56.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:55 smithi023 ceph-mon[25869]: pgmap v97: 65 pgs: 32 unknown, 24 creating+peering, 9 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 511 B/s wr, 0 op/s 2024-03-24T05:29:56.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:55 smithi023 ceph-mon[25869]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi023=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:29:56.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:55 smithi023 ceph-mon[25869]: Saving service alertmanager spec with placement smithi023=a;count:1 2024-03-24T05:29:56.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:55 smithi023 ceph-mon[25869]: osdmap e26: 3 total, 3 up, 3 in 2024-03-24T05:29:56.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:55 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T05:29:56.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:55 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T05:29:56.850 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:56 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T05:29:56.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:56 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T05:29:56.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:56 smithi143 ceph-mon[30288]: osdmap e27: 3 total, 3 up, 3 in 2024-03-24T05:29:57.053 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:56 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T05:29:57.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:56 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T05:29:57.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:56 smithi063 ceph-mon[30595]: osdmap e27: 3 total, 3 up, 3 in 2024-03-24T05:29:57.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:56 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T05:29:57.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:56 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T05:29:57.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:56 smithi023 ceph-mon[25869]: osdmap e27: 3 total, 3 up, 3 in 2024-03-24T05:29:57.280 INFO:teuthology.orchestra.run.smithi143.stdout:Scheduled grafana update... 2024-03-24T05:29:57.882 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:57 smithi143 ceph-mon[30288]: pgmap v100: 97 pgs: 23 unknown, 12 creating+peering, 62 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-24T05:29:57.882 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:57 smithi143 ceph-mon[30288]: osdmap e28: 3 total, 3 up, 3 in 2024-03-24T05:29:57.882 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:57 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T05:29:57.882 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:57 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T05:29:57.882 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:57 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:57.908 DEBUG:teuthology.orchestra.run.smithi063:grafana.a> sudo journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@grafana.a.service 2024-03-24T05:29:57.911 INFO:tasks.cephadm:Setting up client nodes... 2024-03-24T05:29:57.912 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T05:29:58.053 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:57 smithi063 ceph-mon[30595]: pgmap v100: 97 pgs: 23 unknown, 12 creating+peering, 62 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-24T05:29:58.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:57 smithi063 ceph-mon[30595]: osdmap e28: 3 total, 3 up, 3 in 2024-03-24T05:29:58.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:57 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T05:29:58.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:57 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T05:29:58.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:57 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:58.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:57 smithi023 ceph-mon[25869]: pgmap v100: 97 pgs: 23 unknown, 12 creating+peering, 62 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-24T05:29:58.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:57 smithi023 ceph-mon[25869]: osdmap e28: 3 total, 3 up, 3 in 2024-03-24T05:29:58.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:57 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T05:29:58.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:57 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T05:29:58.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:57 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:29:58.159 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:29:59.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:58 smithi063 ceph-mon[30595]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi063=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:29:59.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:58 smithi063 ceph-mon[30595]: Saving service grafana spec with placement smithi063=a;count:1 2024-03-24T05:29:59.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:58 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T05:29:59.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:58 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T05:29:59.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:58 smithi063 ceph-mon[30595]: osdmap e29: 3 total, 3 up, 3 in 2024-03-24T05:29:59.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:58 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T05:29:59.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:58 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T05:29:59.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:58 smithi143 ceph-mon[30288]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi063=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:29:59.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:58 smithi143 ceph-mon[30288]: Saving service grafana spec with placement smithi063=a;count:1 2024-03-24T05:29:59.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:58 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T05:29:59.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:58 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T05:29:59.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:58 smithi143 ceph-mon[30288]: osdmap e29: 3 total, 3 up, 3 in 2024-03-24T05:29:59.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:58 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T05:29:59.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:58 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T05:29:59.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:58 smithi023 ceph-mon[25869]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi063=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:29:59.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:58 smithi023 ceph-mon[25869]: Saving service grafana spec with placement smithi063=a;count:1 2024-03-24T05:29:59.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:58 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T05:29:59.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:58 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T05:29:59.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:58 smithi023 ceph-mon[25869]: osdmap e29: 3 total, 3 up, 3 in 2024-03-24T05:29:59.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:58 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T05:29:59.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:58 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T05:29:59.956 INFO:teuthology.orchestra.run.smithi023.stdout:[client.0] 2024-03-24T05:29:59.956 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQBXuv9lj3bxOBAAHfQvsoP9bi/juInMvR7ZCg== 2024-03-24T05:30:00.053 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:59 smithi063 ceph-mon[30595]: pgmap v103: 129 pgs: 22 unknown, 22 creating+peering, 85 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-24T05:30:00.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:59 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T05:30:00.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:59 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T05:30:00.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:59 smithi063 ceph-mon[30595]: osdmap e30: 3 total, 3 up, 3 in 2024-03-24T05:30:00.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:59 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:30:00.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:59 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:30:00.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:29:59 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:30:00.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:59 smithi143 ceph-mon[30288]: pgmap v103: 129 pgs: 22 unknown, 22 creating+peering, 85 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-24T05:30:00.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:59 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T05:30:00.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:59 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T05:30:00.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:59 smithi143 ceph-mon[30288]: osdmap e30: 3 total, 3 up, 3 in 2024-03-24T05:30:00.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:59 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:30:00.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:59 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:30:00.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:29:59 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:30:00.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:59 smithi023 ceph-mon[25869]: pgmap v103: 129 pgs: 22 unknown, 22 creating+peering, 85 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-24T05:30:00.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:59 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/2398603241' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T05:30:00.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:59 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/302698256' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T05:30:00.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:59 smithi023 ceph-mon[25869]: osdmap e30: 3 total, 3 up, 3 in 2024-03-24T05:30:00.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:59 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:30:00.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:59 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:30:00.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:29:59 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:30:00.539 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T05:30:00.539 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-24T05:30:00.540 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T05:30:00.582 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T05:30:00.829 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T05:30:00.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:00 smithi023 ceph-mon[25869]: Deploying daemon prometheus.a on smithi063 2024-03-24T05:30:00.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:00 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/2265469040' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T05:30:00.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:00 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/2265469040' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T05:30:00.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:00 smithi023 ceph-mon[25869]: overall HEALTH_OK 2024-03-24T05:30:00.949 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:00 smithi063 ceph-mon[30595]: Deploying daemon prometheus.a on smithi063 2024-03-24T05:30:00.949 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:00 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/2265469040' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T05:30:00.950 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:00 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/2265469040' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T05:30:00.950 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:00 smithi063 ceph-mon[30595]: overall HEALTH_OK 2024-03-24T05:30:01.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:00 smithi143 ceph-mon[30288]: Deploying daemon prometheus.a on smithi063 2024-03-24T05:30:01.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:00 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/2265469040' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T05:30:01.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:00 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/2265469040' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T05:30:01.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:00 smithi143 ceph-mon[30288]: overall HEALTH_OK 2024-03-24T05:30:02.053 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:01 smithi063 ceph-mon[30595]: pgmap v105: 129 pgs: 22 unknown, 22 creating+peering, 85 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.3 KiB/s rd, 454 B/s wr, 2 op/s 2024-03-24T05:30:02.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:01 smithi143 ceph-mon[30288]: pgmap v105: 129 pgs: 22 unknown, 22 creating+peering, 85 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.3 KiB/s rd, 454 B/s wr, 2 op/s 2024-03-24T05:30:02.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:01 smithi023 ceph-mon[25869]: pgmap v105: 129 pgs: 22 unknown, 22 creating+peering, 85 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.3 KiB/s rd, 454 B/s wr, 2 op/s 2024-03-24T05:30:03.852 INFO:teuthology.orchestra.run.smithi063.stdout:[client.1] 2024-03-24T05:30:03.852 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBbuv9lE8CbMhAAWHjFaF3IwtdqB2Jqh49Ejg== 2024-03-24T05:30:04.053 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:03 smithi063 ceph-mon[30595]: pgmap v106: 129 pgs: 10 creating+peering, 119 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 75 KiB/s rd, 3.8 KiB/s wr, 167 op/s 2024-03-24T05:30:04.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:03 smithi063 ceph-mon[30595]: from='client.? 172.21.15.63:0/2862567615' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T05:30:04.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:03 smithi063 ceph-mon[30595]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T05:30:04.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:03 smithi063 ceph-mon[30595]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T05:30:04.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:03 smithi143 ceph-mon[30288]: pgmap v106: 129 pgs: 10 creating+peering, 119 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 75 KiB/s rd, 3.8 KiB/s wr, 167 op/s 2024-03-24T05:30:04.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:03 smithi143 ceph-mon[30288]: from='client.? 172.21.15.63:0/2862567615' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T05:30:04.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:03 smithi143 ceph-mon[30288]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T05:30:04.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:03 smithi143 ceph-mon[30288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T05:30:04.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:03 smithi023 ceph-mon[25869]: pgmap v106: 129 pgs: 10 creating+peering, 119 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 75 KiB/s rd, 3.8 KiB/s wr, 167 op/s 2024-03-24T05:30:04.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:03 smithi023 ceph-mon[25869]: from='client.? 172.21.15.63:0/2862567615' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T05:30:04.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:03 smithi023 ceph-mon[25869]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T05:30:04.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:03 smithi023 ceph-mon[25869]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T05:30:04.737 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-24T05:30:04.737 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-24T05:30:04.738 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-24T05:30:04.781 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T05:30:05.025 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T05:30:06.021 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:05 smithi143 ceph-mon[30288]: pgmap v107: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 96 KiB/s rd, 4.7 KiB/s wr, 213 op/s 2024-03-24T05:30:06.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:05 smithi063 ceph-mon[30595]: pgmap v107: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 96 KiB/s rd, 4.7 KiB/s wr, 213 op/s 2024-03-24T05:30:06.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:05 smithi023 ceph-mon[25869]: pgmap v107: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 96 KiB/s rd, 4.7 KiB/s wr, 213 op/s 2024-03-24T05:30:06.773 INFO:teuthology.orchestra.run.smithi143.stdout:[client.2] 2024-03-24T05:30:06.774 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQBeuv9lsWv3LRAAXhCAwj+pH2YPIztfEguNAQ== 2024-03-24T05:30:07.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:06 smithi143 ceph-mon[30288]: from='client.? 172.21.15.143:0/4224026285' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T05:30:07.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:06 smithi143 ceph-mon[30288]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T05:30:07.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:06 smithi143 ceph-mon[30288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T05:30:07.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:06 smithi063 ceph-mon[30595]: from='client.? 172.21.15.143:0/4224026285' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T05:30:07.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:06 smithi063 ceph-mon[30595]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T05:30:07.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:06 smithi063 ceph-mon[30595]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T05:30:07.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:06 smithi023 ceph-mon[25869]: from='client.? 172.21.15.143:0/4224026285' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T05:30:07.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:06 smithi023 ceph-mon[25869]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T05:30:07.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:06 smithi023 ceph-mon[25869]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T05:30:07.404 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T05:30:07.404 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-03-24T05:30:07.404 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-24T05:30:07.445 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T05:30:07.446 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-24T05:30:07.446 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-24T05:30:07.699 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:30:08.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:07 smithi023 ceph-mon[25869]: pgmap v108: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 84 KiB/s rd, 4 KiB/s wr, 187 op/s 2024-03-24T05:30:08.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:07 smithi063 ceph-mon[30595]: pgmap v108: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 84 KiB/s rd, 4 KiB/s wr, 187 op/s 2024-03-24T05:30:08.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:07 smithi143 ceph-mon[30288]: pgmap v108: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 84 KiB/s rd, 4 KiB/s wr, 187 op/s 2024-03-24T05:30:09.632 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:30:10.226 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:10 smithi023 ceph-mon[25869]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 71 KiB/s rd, 3.4 KiB/s wr, 159 op/s 2024-03-24T05:30:10.226 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:10 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/3804263400' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T05:30:10.252 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1460774210},{"type":"v1","addr":"172.21.15.23:6801","nonce":1460774210}]},"active_addr":"172.21.15.23:6801/1460774210","active_change":"2024-03-24T05:27:10.172573+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.23:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":1599363014}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":2461598639}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":1669425486}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":1255539830}]}]} 2024-03-24T05:30:10.259 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-24T05:30:10.259 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T05:30:10.260 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T05:30:10.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:10 smithi063 ceph-mon[30595]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 71 KiB/s rd, 3.4 KiB/s wr, 159 op/s 2024-03-24T05:30:10.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:10 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/3804263400' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T05:30:10.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:10 smithi143 ceph-mon[30288]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 71 KiB/s rd, 3.4 KiB/s wr, 159 op/s 2024-03-24T05:30:10.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:10 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/3804263400' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T05:30:10.497 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:30:11.256 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 systemd[1]: Starting Ceph prometheus.a for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:30:11.498 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:11 smithi023 ceph-mon[25869]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 62 KiB/s rd, 3.0 KiB/s wr, 138 op/s 2024-03-24T05:30:11.499 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:11 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:30:11.521 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-mon[30595]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 62 KiB/s rd, 3.0 KiB/s wr, 138 op/s 2024-03-24T05:30:11.521 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:30:11.521 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 podman[41741]: 2024-03-24 05:30:11.16057821 +0000 UTC m=+0.019017924 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-24T05:30:11.521 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 podman[41741]: 2024-03-24 05:30:11.258701097 +0000 UTC m=+0.117140835 container create d84f0a1ee3acc8ba8bf428ba2438e998ae74fa4aab84e906118de5695e424a51 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T05:30:11.521 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 podman[41741]: 2024-03-24 05:30:11.494111981 +0000 UTC m=+0.352551687 container init d84f0a1ee3acc8ba8bf428ba2438e998ae74fa4aab84e906118de5695e424a51 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T05:30:11.521 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 podman[41741]: 2024-03-24 05:30:11.499117087 +0000 UTC m=+0.357556780 container start d84f0a1ee3acc8ba8bf428ba2438e998ae74fa4aab84e906118de5695e424a51 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T05:30:11.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:11 smithi143 ceph-mon[30288]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 62 KiB/s rd, 3.0 KiB/s wr, 138 op/s 2024-03-24T05:30:11.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:11 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:30:11.804 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:30:11.522Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-24T05:30:11.804 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:30:11.522Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-24T05:30:11.804 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:30:11.522Z caller=main.go:570 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi063 (none))" 2024-03-24T05:30:11.805 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:30:11.522Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-24T05:30:11.805 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:30:11.522Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-24T05:30:11.805 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:30:11.523Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-24T05:30:11.805 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:30:11.524Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-24T05:30:11.805 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:30:11.525Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-24T05:30:11.805 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:30:11.525Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-24T05:30:11.805 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:30:11.527Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-24T05:30:11.805 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:30:11.527Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.78µs 2024-03-24T05:30:11.806 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:30:11.527Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-24T05:30:11.806 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:30:11.527Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-24T05:30:11.806 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:30:11.527Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.252µs wal_replay_duration=207.92µs wbl_replay_duration=114ns total_replay_duration=244.797µs 2024-03-24T05:30:11.806 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:30:11.528Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-24T05:30:11.806 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:30:11.528Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-24T05:30:11.806 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:30:11.528Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-24T05:30:11.806 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:30:11.545Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.304046ms db_storage=672ns remote_storage=1.036µs web_handler=335ns query_engine=556ns scrape=391.996µs scrape_sd=73.145µs notify=740ns notify_sd=1.512µs rules=16.649607ms tracing=6.339µs 2024-03-24T05:30:11.806 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:30:11.545Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-24T05:30:11.806 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:30:11.545Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-24T05:30:11.806 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 bash[41741]: d84f0a1ee3acc8ba8bf428ba2438e998ae74fa4aab84e906118de5695e424a51 2024-03-24T05:30:11.807 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:30:11 smithi063 systemd[1]: Started Ceph prometheus.a for e2aa55c2-e99e-11ee-95cd-87774f69a715. 2024-03-24T05:30:12.235 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:30:12.235 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"fsid":"e2aa55c2-e99e-11ee-95cd-87774f69a715","created":"2024-03-24T05:26:10.742267+0000","modified":"2024-03-24T05:29:58.718174+0000","last_up_change":"2024-03-24T05:29:35.130293+0000","last_in_change":"2024-03-24T05:29:16.039751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T05:29:36.514453+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-24T05:29:49.970560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-24T05:29:51.992073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-24T05:29:53.952664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-24T05:29:56.020723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38483f5a-088a-4caa-9228-5502305de04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":489464929},{"type":"v1","addr":"172.21.15.23:6803","nonce":489464929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":489464929},{"type":"v1","addr":"172.21.15.23:6805","nonce":489464929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":489464929},{"type":"v1","addr":"172.21.15.23:6809","nonce":489464929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":489464929},{"type":"v1","addr":"172.21.15.23:6807","nonce":489464929}]},"public_addr":"172.21.15.23:6803/489464929","cluster_addr":"172.21.15.23:6805/489464929","heartbeat_back_addr":"172.21.15.23:6809/489464929","heartbeat_front_addr":"172.21.15.23:6807/489464929","state":["exists","up"]},{"osd":1,"uuid":"bb781706-c5f9-4f47-8f0b-5784a95159a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1403713781},{"type":"v1","addr":"172.21.15.63:6801","nonce":1403713781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1403713781},{"type":"v1","addr":"172.21.15.63:6803","nonce":1403713781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1403713781},{"type":"v1","addr":"172.21.15.63:6807","nonce":1403713781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1403713781},{"type":"v1","addr":"172.21.15.63:6805","nonce":1403713781}]},"public_addr":"172.21.15.63:6801/1403713781","cluster_addr":"172.21.15.63:6803/1403713781","heartbeat_back_addr":"172.21.15.63:6807/1403713781","heartbeat_front_addr":"172.21.15.63:6805/1403713781","state":["exists","up"]},{"osd":2,"uuid":"374d416e-cb02-420f-995d-0e6e035dd239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":573156693},{"type":"v1","addr":"172.21.15.143:6801","nonce":573156693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":573156693},{"type":"v1","addr":"172.21.15.143:6803","nonce":573156693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":573156693},{"type":"v1","addr":"172.21.15.143:6807","nonce":573156693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":573156693},{"type":"v1","addr":"172.21.15.143:6805","nonce":573156693}]},"public_addr":"172.21.15.143:6801/573156693","cluster_addr":"172.21.15.143:6803/573156693","heartbeat_back_addr":"172.21.15.143:6807/573156693","heartbeat_front_addr":"172.21.15.143:6805/573156693","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T05:28:44.538609+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T05:29:08.577419+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T05:29:32.791088+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/477877701":"2024-03-25T05:27:10.172457+0000","172.21.15.23:0/3117010616":"2024-03-25T05:27:10.172457+0000","172.21.15.23:0/2279719634":"2024-03-25T05:26:43.320335+0000","172.21.15.23:6800/3490012969":"2024-03-25T05:26:43.320335+0000","172.21.15.23:6800/477877701":"2024-03-25T05:27:10.172457+0000","172.21.15.23:0/828298596":"2024-03-25T05:27:10.172457+0000","172.21.15.23:0/1184189117":"2024-03-25T05:27:10.172457+0000","172.21.15.23:0/1787617234":"2024-03-25T05:26:43.320335+0000","172.21.15.23:6801/3490012969":"2024-03-25T05:26:43.320335+0000","172.21.15.23:0/1789110770":"2024-03-25T05:26:43.320335+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T05:30:12.830 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:12 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: ignoring --setuser ceph since I am not root 2024-03-24T05:30:12.831 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:12 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: ignoring --setgroup ceph since I am not root 2024-03-24T05:30:12.831 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:12 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:30:12.831 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:12 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:30:12.831 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:12 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:30:12.831 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:12 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T05:30:12.831 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:12 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/2410570933' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T05:30:12.856 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T05:30:12.857 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T05:30:13.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:12 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:30:13.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:12 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:30:13.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:12 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:30:13.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:12 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T05:30:13.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:12 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/2410570933' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T05:30:13.054 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:12 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: ignoring --setuser ceph since I am not root 2024-03-24T05:30:13.054 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:12 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: ignoring --setgroup ceph since I am not root 2024-03-24T05:30:13.055 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:12 smithi063 ceph-mgr[32075]: -- 172.21.15.63:0/3831650566 <== mon.2 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5587747a41a0 con 0x55877477b000 2024-03-24T05:30:13.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:12 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:30:13.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:12 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:30:13.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:12 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' 2024-03-24T05:30:13.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:12 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T05:30:13.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:12 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/2410570933' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T05:30:13.058 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:12 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: ignoring --setuser ceph since I am not root 2024-03-24T05:30:13.058 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:12 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: ignoring --setgroup ceph since I am not root 2024-03-24T05:30:13.059 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:12 smithi143 ceph-mgr[32380]: -- 172.21.15.143:0/1097296117 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5561de9201a0 con 0x5561de8f6800 2024-03-24T05:30:13.102 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:30:13.553 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:13 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:13.231+0000 7fd058e5d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T05:30:13.553 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:13 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:13.327+0000 7fd058e5d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T05:30:13.558 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:13 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:13.266+0000 7f017a7c0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T05:30:13.558 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:13 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:13.368+0000 7f017a7c0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T05:30:13.558 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:13 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:13.215+0000 7f0aa4077200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T05:30:13.558 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:13 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:13.308+0000 7f0aa4077200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T05:30:13.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:13 smithi023 ceph-mon[25869]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T05:30:13.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:13 smithi023 ceph-mon[25869]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-24T05:30:13.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:13 smithi143 ceph-mon[30288]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T05:30:13.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:13 smithi143 ceph-mon[30288]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-24T05:30:13.981 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:13 smithi063 ceph-mon[30595]: from='mgr.14152 172.21.15.23:0/986448862' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T05:30:13.981 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:13 smithi063 ceph-mon[30595]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-24T05:30:14.194 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:14 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:14.044+0000 7f017a7c0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T05:30:14.194 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:14 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:14.141+0000 7f017a7c0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T05:30:14.272 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:13 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:13.942+0000 7f0aa4077200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T05:30:14.272 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:14 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:14.034+0000 7f0aa4077200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T05:30:14.272 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:14 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:14.176+0000 7f0aa4077200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T05:30:14.272 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:14 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:14.271+0000 7f0aa4077200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T05:30:14.303 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:13 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:13.981+0000 7fd058e5d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T05:30:14.304 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:14 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:14.076+0000 7fd058e5d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T05:30:14.304 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:14 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:14.219+0000 7fd058e5d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T05:30:14.477 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:14 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:14.289+0000 7f017a7c0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T05:30:14.477 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:14 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:14.388+0000 7f017a7c0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T05:30:14.477 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:14 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:14.476+0000 7f017a7c0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T05:30:14.557 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:14 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:14.353+0000 7f0aa4077200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T05:30:14.666 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:14 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:14.316+0000 7fd058e5d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T05:30:14.666 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:14 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:14.400+0000 7fd058e5d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T05:30:14.667 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:14 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:14.666+0000 7fd058e5d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T05:30:14.904 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:14 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:14.616+0000 7f0aa4077200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T05:30:14.904 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:14 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:14.723+0000 7f0aa4077200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T05:30:14.904 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:14 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:14.809+0000 7f0aa4077200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T05:30:14.904 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:14 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:14.903+0000 7f0aa4077200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T05:30:14.911 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:30:14.911 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"fsid":"e2aa55c2-e99e-11ee-95cd-87774f69a715","created":"2024-03-24T05:26:10.742267+0000","modified":"2024-03-24T05:29:58.718174+0000","last_up_change":"2024-03-24T05:29:35.130293+0000","last_in_change":"2024-03-24T05:29:16.039751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T05:29:36.514453+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-24T05:29:49.970560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-24T05:29:51.992073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-24T05:29:53.952664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-24T05:29:56.020723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38483f5a-088a-4caa-9228-5502305de04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":489464929},{"type":"v1","addr":"172.21.15.23:6803","nonce":489464929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":489464929},{"type":"v1","addr":"172.21.15.23:6805","nonce":489464929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":489464929},{"type":"v1","addr":"172.21.15.23:6809","nonce":489464929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":489464929},{"type":"v1","addr":"172.21.15.23:6807","nonce":489464929}]},"public_addr":"172.21.15.23:6803/489464929","cluster_addr":"172.21.15.23:6805/489464929","heartbeat_back_addr":"172.21.15.23:6809/489464929","heartbeat_front_addr":"172.21.15.23:6807/489464929","state":["exists","up"]},{"osd":1,"uuid":"bb781706-c5f9-4f47-8f0b-5784a95159a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1403713781},{"type":"v1","addr":"172.21.15.63:6801","nonce":1403713781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1403713781},{"type":"v1","addr":"172.21.15.63:6803","nonce":1403713781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1403713781},{"type":"v1","addr":"172.21.15.63:6807","nonce":1403713781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1403713781},{"type":"v1","addr":"172.21.15.63:6805","nonce":1403713781}]},"public_addr":"172.21.15.63:6801/1403713781","cluster_addr":"172.21.15.63:6803/1403713781","heartbeat_back_addr":"172.21.15.63:6807/1403713781","heartbeat_front_addr":"172.21.15.63:6805/1403713781","state":["exists","up"]},{"osd":2,"uuid":"374d416e-cb02-420f-995d-0e6e035dd239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":573156693},{"type":"v1","addr":"172.21.15.143:6801","nonce":573156693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":573156693},{"type":"v1","addr":"172.21.15.143:6803","nonce":573156693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":573156693},{"type":"v1","addr":"172.21.15.143:6807","nonce":573156693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":573156693},{"type":"v1","addr":"172.21.15.143:6805","nonce":573156693}]},"public_addr":"172.21.15.143:6801/573156693","cluster_addr":"172.21.15.143:6803/573156693","heartbeat_back_addr":"172.21.15.143:6807/573156693","heartbeat_front_addr":"172.21.15.143:6805/573156693","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T05:28:44.538609+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T05:29:08.577419+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T05:29:32.791088+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/477877701":"2024-03-25T05:27:10.172457+0000","172.21.15.23:0/3117010616":"2024-03-25T05:27:10.172457+0000","172.21.15.23:0/2279719634":"2024-03-25T05:26:43.320335+0000","172.21.15.23:6800/3490012969":"2024-03-25T05:26:43.320335+0000","172.21.15.23:6800/477877701":"2024-03-25T05:27:10.172457+0000","172.21.15.23:0/828298596":"2024-03-25T05:27:10.172457+0000","172.21.15.23:0/1184189117":"2024-03-25T05:27:10.172457+0000","172.21.15.23:0/1787617234":"2024-03-25T05:26:43.320335+0000","172.21.15.23:6801/3490012969":"2024-03-25T05:26:43.320335+0000","172.21.15.23:0/1789110770":"2024-03-25T05:26:43.320335+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T05:30:14.962 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:14 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:14.778+0000 7fd058e5d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T05:30:14.962 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:14 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:14.866+0000 7fd058e5d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T05:30:14.962 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:14 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:14.962+0000 7fd058e5d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T05:30:15.065 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:14 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/305850705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T05:30:15.065 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:14 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:14.755+0000 7f017a7c0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T05:30:15.065 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:14 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:14.868+0000 7f017a7c0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T05:30:15.065 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:14 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:14.966+0000 7f017a7c0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T05:30:15.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:14 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/305850705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T05:30:15.304 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:15 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:15.140+0000 7fd058e5d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T05:30:15.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:14 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/305850705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T05:30:15.307 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:15 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:15.080+0000 7f0aa4077200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T05:30:15.329 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:15 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:15.065+0000 7f017a7c0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T05:30:15.329 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:15 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:15.247+0000 7f017a7c0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T05:30:15.545 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T05:30:15.545 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T05:30:15.545 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T05:30:15.851 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:30:15.852 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:30:15.852 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:30:16.307 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:16 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:16.030+0000 7f0aa4077200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T05:30:16.308 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:16 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:16.160+0000 7f0aa4077200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T05:30:16.553 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:16 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:16.120+0000 7fd058e5d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T05:30:16.553 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:16 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:16.252+0000 7fd058e5d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T05:30:16.557 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:16 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:16.306+0000 7f017a7c0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T05:30:16.557 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:16 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:16.444+0000 7f017a7c0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T05:30:16.776 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:16 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:16.492+0000 7f0aa4077200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T05:30:16.776 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:16 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:16.583+0000 7f0aa4077200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T05:30:16.777 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:16 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:16.777+0000 7f0aa4077200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T05:30:16.829 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:16 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:16.792+0000 7f017a7c0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T05:30:16.879 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:16 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:16.588+0000 7fd058e5d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T05:30:16.879 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:16 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:16.679+0000 7fd058e5d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T05:30:17.057 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:16 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:16.961+0000 7f0aa4077200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T05:30:17.282 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:16 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:16.886+0000 7f017a7c0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T05:30:17.282 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:17 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:17.089+0000 7f017a7c0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T05:30:17.282 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:17 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:17.281+0000 7f017a7c0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T05:30:17.303 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:16 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:16.879+0000 7fd058e5d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T05:30:17.303 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:17 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:17.067+0000 7fd058e5d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T05:30:17.501 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:17 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:17.242+0000 7f0aa4077200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T05:30:17.501 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:17 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:17.501+0000 7f0aa4077200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T05:30:17.615 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:17 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:17.356+0000 7fd058e5d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T05:30:17.807 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:17 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:17.734+0000 7f0aa4077200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T05:30:17.831 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:17 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:17.581+0000 7f017a7c0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T05:30:18.053 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:17 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:17.615+0000 7fd058e5d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T05:30:18.053 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:17 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:17.858+0000 7fd058e5d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T05:30:18.089 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:17 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:17.846+0000 7f017a7c0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T05:30:18.457 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:18 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:18.089+0000 7f017a7c0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T05:30:18.807 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:18 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:18.521+0000 7f0aa4077200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T05:30:18.986 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:18 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:18.943+0000 7f017a7c0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T05:30:19.053 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:18 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:18.679+0000 7fd058e5d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T05:30:19.401 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:19 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:19.090+0000 7f0aa4077200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T05:30:19.401 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:19 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:19.187+0000 7f0aa4077200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T05:30:19.401 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:19 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:19.401+0000 7f0aa4077200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T05:30:19.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:19 smithi063 ceph-mon[30595]: Standby manager daemon z restarted 2024-03-24T05:30:19.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:19 smithi063 ceph-mon[30595]: Standby manager daemon z started 2024-03-24T05:30:19.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:19 smithi063 ceph-mon[30595]: from='mgr.? 172.21.15.143:0/587419579' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T05:30:19.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:19 smithi063 ceph-mon[30595]: from='mgr.? 172.21.15.143:0/587419579' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T05:30:19.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:19 smithi063 ceph-mon[30595]: from='mgr.? 172.21.15.143:0/587419579' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T05:30:19.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:19 smithi063 ceph-mon[30595]: from='mgr.? 172.21.15.143:0/587419579' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T05:30:19.555 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:19 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:19.281+0000 7fd058e5d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T05:30:19.555 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:19 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:19.382+0000 7fd058e5d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T05:30:19.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:19 smithi023 ceph-mon[25869]: Standby manager daemon z restarted 2024-03-24T05:30:19.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:19 smithi023 ceph-mon[25869]: Standby manager daemon z started 2024-03-24T05:30:19.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:19 smithi023 ceph-mon[25869]: from='mgr.? 172.21.15.143:0/587419579' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T05:30:19.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:19 smithi023 ceph-mon[25869]: from='mgr.? 172.21.15.143:0/587419579' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T05:30:19.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:19 smithi023 ceph-mon[25869]: from='mgr.? 172.21.15.143:0/587419579' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T05:30:19.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:19 smithi023 ceph-mon[25869]: from='mgr.? 172.21.15.143:0/587419579' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T05:30:19.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:19 smithi143 ceph-mon[30288]: Standby manager daemon z restarted 2024-03-24T05:30:19.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:19 smithi143 ceph-mon[30288]: Standby manager daemon z started 2024-03-24T05:30:19.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:19 smithi143 ceph-mon[30288]: from='mgr.? 172.21.15.143:0/587419579' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T05:30:19.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:19 smithi143 ceph-mon[30288]: from='mgr.? 172.21.15.143:0/587419579' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T05:30:19.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:19 smithi143 ceph-mon[30288]: from='mgr.? 172.21.15.143:0/587419579' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T05:30:19.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:19 smithi143 ceph-mon[30288]: from='mgr.? 172.21.15.143:0/587419579' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T05:30:19.809 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:19 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: [24/Mar/2024:05:30:19] ENGINE Bus STARTING 2024-03-24T05:30:19.809 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:19 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: CherryPy Checker: 2024-03-24T05:30:19.809 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:19 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: The Application mounted at '' has an empty config. 2024-03-24T05:30:19.809 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:19 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: 2024-03-24T05:30:19.809 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:19 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: [24/Mar/2024:05:30:19] ENGINE Serving on http://:::9283 2024-03-24T05:30:19.809 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:30:19 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z[32356]: [24/Mar/2024:05:30:19] ENGINE Bus STARTED 2024-03-24T05:30:19.916 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:19 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:19.585+0000 7f017a7c0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T05:30:19.916 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:19 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:19.689+0000 7f017a7c0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T05:30:19.916 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:19 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:19.916+0000 7f017a7c0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T05:30:20.053 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:19 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:19.603+0000 7fd058e5d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T05:30:20.054 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:19 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: [24/Mar/2024:05:30:19] ENGINE Bus STARTING 2024-03-24T05:30:20.054 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:19 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: CherryPy Checker: 2024-03-24T05:30:20.054 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:19 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: The Application mounted at '' has an empty config. 2024-03-24T05:30:20.054 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:19 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: 2024-03-24T05:30:20.054 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:19 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: [24/Mar/2024:05:30:19] ENGINE Serving on http://:::9283 2024-03-24T05:30:20.054 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:30:19 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x[32052]: [24/Mar/2024:05:30:19] ENGINE Bus STARTED 2024-03-24T05:30:20.059 INFO:teuthology.orchestra.run.smithi023.stdout:34359738389 2024-03-24T05:30:20.060 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T05:30:20.272 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: [24/Mar/2024:05:30:20] ENGINE Bus STARTING 2024-03-24T05:30:20.272 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: CherryPy Checker: 2024-03-24T05:30:20.272 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: The Application mounted at '' has an empty config. 2024-03-24T05:30:20.272 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:30:20.272 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: [24/Mar/2024:05:30:20] ENGINE Serving on http://:::9283 2024-03-24T05:30:20.272 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: [24/Mar/2024:05:30:20] ENGINE Bus STARTED 2024-03-24T05:30:20.363 INFO:teuthology.orchestra.run.smithi023.stdout:55834574864 2024-03-24T05:30:20.364 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T05:30:20.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-24T05:30:20.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: Standby manager daemon x restarted 2024-03-24T05:30:20.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: Standby manager daemon x started 2024-03-24T05:30:20.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.? 172.21.15.63:0/2993479606' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T05:30:20.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.? 172.21.15.63:0/2993479606' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T05:30:20.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.? 172.21.15.63:0/2993479606' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T05:30:20.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.? 172.21.15.63:0/2993479606' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T05:30:20.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: Active manager daemon y restarted 2024-03-24T05:30:20.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: Activating manager daemon y 2024-03-24T05:30:20.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: osdmap e31: 3 total, 3 up, 3 in 2024-03-24T05:30:20.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: mgrmap e19: y(active, starting, since 0.0124428s), standbys: x, z 2024-03-24T05:30:20.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T05:30:20.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:30:20.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:30:20.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T05:30:20.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T05:30:20.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T05:30:20.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T05:30:20.582 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T05:30:20.582 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T05:30:20.582 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T05:30:20.582 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T05:30:20.582 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T05:30:20.582 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: Manager daemon y is now available 2024-03-24T05:30:20.582 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:30:20.582 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:30:20.582 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:20.583 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T05:30:20.583 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T05:30:20.583 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:20 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T05:30:20.731 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:30:20.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-24T05:30:20.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: Standby manager daemon x restarted 2024-03-24T05:30:20.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: Standby manager daemon x started 2024-03-24T05:30:20.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.? 172.21.15.63:0/2993479606' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T05:30:20.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.? 172.21.15.63:0/2993479606' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T05:30:20.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.? 172.21.15.63:0/2993479606' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T05:30:20.805 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.? 172.21.15.63:0/2993479606' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T05:30:20.805 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: Active manager daemon y restarted 2024-03-24T05:30:20.805 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: Activating manager daemon y 2024-03-24T05:30:20.805 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: osdmap e31: 3 total, 3 up, 3 in 2024-03-24T05:30:20.805 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: mgrmap e19: y(active, starting, since 0.0124428s), standbys: x, z 2024-03-24T05:30:20.805 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T05:30:20.805 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:30:20.805 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:30:20.805 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T05:30:20.805 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T05:30:20.806 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T05:30:20.806 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T05:30:20.806 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T05:30:20.806 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T05:30:20.806 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T05:30:20.806 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T05:30:20.806 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T05:30:20.806 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: Manager daemon y is now available 2024-03-24T05:30:20.806 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:30:20.807 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:30:20.807 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:20.807 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T05:30:20.807 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T05:30:20.807 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:20 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T05:30:20.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-24T05:30:20.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: Standby manager daemon x restarted 2024-03-24T05:30:20.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: Standby manager daemon x started 2024-03-24T05:30:20.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.? 172.21.15.63:0/2993479606' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T05:30:20.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.? 172.21.15.63:0/2993479606' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T05:30:20.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.? 172.21.15.63:0/2993479606' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T05:30:20.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.? 172.21.15.63:0/2993479606' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T05:30:20.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: Active manager daemon y restarted 2024-03-24T05:30:20.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: Activating manager daemon y 2024-03-24T05:30:20.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: osdmap e31: 3 total, 3 up, 3 in 2024-03-24T05:30:20.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: mgrmap e19: y(active, starting, since 0.0124428s), standbys: x, z 2024-03-24T05:30:20.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T05:30:20.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T05:30:20.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T05:30:20.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T05:30:20.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T05:30:20.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T05:30:20.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T05:30:20.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T05:30:20.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T05:30:20.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T05:30:20.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T05:30:20.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T05:30:20.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: Manager daemon y is now available 2024-03-24T05:30:20.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:30:20.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T05:30:20.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:20.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T05:30:20.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T05:30:20.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:20 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T05:30:21.612 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:30:22.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:21 smithi023 ceph-mon[25869]: mgrmap e20: y(active, since 1.01545s), standbys: x, z 2024-03-24T05:30:22.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:21 smithi023 ceph-mon[25869]: [24/Mar/2024:05:30:21] ENGINE Bus STARTING 2024-03-24T05:30:22.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:21 smithi023 ceph-mon[25869]: [24/Mar/2024:05:30:21] ENGINE Serving on https://172.21.15.23:7150 2024-03-24T05:30:22.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:21 smithi023 ceph-mon[25869]: [24/Mar/2024:05:30:21] ENGINE Serving on http://172.21.15.23:8765 2024-03-24T05:30:22.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:21 smithi023 ceph-mon[25869]: [24/Mar/2024:05:30:21] ENGINE Bus STARTED 2024-03-24T05:30:22.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:21 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:22.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:21 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:22.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:21 smithi063 ceph-mon[30595]: mgrmap e20: y(active, since 1.01545s), standbys: x, z 2024-03-24T05:30:22.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:21 smithi063 ceph-mon[30595]: [24/Mar/2024:05:30:21] ENGINE Bus STARTING 2024-03-24T05:30:22.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:21 smithi063 ceph-mon[30595]: [24/Mar/2024:05:30:21] ENGINE Serving on https://172.21.15.23:7150 2024-03-24T05:30:22.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:21 smithi063 ceph-mon[30595]: [24/Mar/2024:05:30:21] ENGINE Serving on http://172.21.15.23:8765 2024-03-24T05:30:22.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:21 smithi063 ceph-mon[30595]: [24/Mar/2024:05:30:21] ENGINE Bus STARTED 2024-03-24T05:30:22.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:21 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:22.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:21 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:22.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:21 smithi143 ceph-mon[30288]: mgrmap e20: y(active, since 1.01545s), standbys: x, z 2024-03-24T05:30:22.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:21 smithi143 ceph-mon[30288]: [24/Mar/2024:05:30:21] ENGINE Bus STARTING 2024-03-24T05:30:22.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:21 smithi143 ceph-mon[30288]: [24/Mar/2024:05:30:21] ENGINE Serving on https://172.21.15.23:7150 2024-03-24T05:30:22.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:21 smithi143 ceph-mon[30288]: [24/Mar/2024:05:30:21] ENGINE Serving on http://172.21.15.23:8765 2024-03-24T05:30:22.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:21 smithi143 ceph-mon[30288]: [24/Mar/2024:05:30:21] ENGINE Bus STARTED 2024-03-24T05:30:22.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:21 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:22.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:21 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:22.606 INFO:teuthology.orchestra.run.smithi023.stdout:77309411339 2024-03-24T05:30:22.606 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T05:30:23.443 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:23 smithi023 ceph-mon[25869]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:23.443 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:23 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:23.443 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:23 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:23.443 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:23 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:23.443 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:23 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:23.444 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:23 smithi023 ceph-mon[25869]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-24T05:30:23.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:23 smithi063 ceph-mon[30595]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:23.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:23 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:23.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:23 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:23.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:23 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:23.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:23 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:23.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:23 smithi063 ceph-mon[30595]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-24T05:30:23.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:23 smithi143 ceph-mon[30288]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:23.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:23 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:23.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:23 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:23.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:23 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:23.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:23 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:23.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:23 smithi143 ceph-mon[30288]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-24T05:30:24.651 INFO:teuthology.orchestra.run.smithi023.stdout:34359738390 2024-03-24T05:30:24.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:24 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:24.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:24 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:24.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:24 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T05:30:24.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:24 smithi023 ceph-mon[25869]: Adjusting osd_memory_target on smithi063 to 16064M 2024-03-24T05:30:24.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:24 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:24.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:24 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:24.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:24 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:24.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:24 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T05:30:24.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:24 smithi063 ceph-mon[30595]: Adjusting osd_memory_target on smithi063 to 16064M 2024-03-24T05:30:24.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:24 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:24.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:24 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:24.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:24 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:24.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:24 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T05:30:24.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:24 smithi143 ceph-mon[30288]: Adjusting osd_memory_target on smithi063 to 16064M 2024-03-24T05:30:24.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:24 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:25.270 INFO:teuthology.orchestra.run.smithi023.stdout:55834574865 2024-03-24T05:30:25.421 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738389 got 34359738390 for osd.0 2024-03-24T05:30:25.421 DEBUG:teuthology.parallel:result is None 2024-03-24T05:30:25.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:25 smithi023 ceph-mon[25869]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:25.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:25 smithi023 ceph-mon[25869]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-24T05:30:25.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:25 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/3820256914' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T05:30:25.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:25 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/652617400' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T05:30:25.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:25 smithi063 ceph-mon[30595]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:25.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:25 smithi063 ceph-mon[30595]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-24T05:30:25.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:25 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/3820256914' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T05:30:25.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:25 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/652617400' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T05:30:25.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:25 smithi143 ceph-mon[30288]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:25.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:25 smithi143 ceph-mon[30288]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-24T05:30:25.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:25 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/3820256914' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T05:30:25.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:25 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/652617400' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T05:30:26.137 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574864 got 55834574865 for osd.1 2024-03-24T05:30:26.137 DEBUG:teuthology.parallel:result is None 2024-03-24T05:30:26.152 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:30:27.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:27 smithi063 ceph-mon[30595]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:27.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:27 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:27.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:27 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:27.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:27 smithi143 ceph-mon[30288]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:27.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:27 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:27.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:27 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:27.816 INFO:teuthology.orchestra.run.smithi023.stdout:77309411339 2024-03-24T05:30:27.816 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:27 smithi023 ceph-mon[25869]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:27.816 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:27 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:27.816 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:27 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:28.079 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:27 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: ::ffff:172.21.15.63 - - [24/Mar/2024:05:30:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T05:30:28.504 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411339 got 77309411339 for osd.2 2024-03-24T05:30:28.504 DEBUG:teuthology.parallel:result is None 2024-03-24T05:30:28.505 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T05:30:28.505 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T05:30:28.787 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:30:28.810 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:28 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:28.810 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:28 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/337553466' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T05:30:28.810 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:28 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:28.810 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:28 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T05:30:28.810 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:28 smithi023 ceph-mon[25869]: Adjusting osd_memory_target on smithi023 to 16064M 2024-03-24T05:30:28.810 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:28 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:28.810 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:28 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:30:28.810 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:28 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:30:28.810 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:28 smithi023 ceph-mon[25869]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-24T05:30:28.810 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:28 smithi023 ceph-mon[25869]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-24T05:30:28.811 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:28 smithi023 ceph-mon[25869]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T05:30:28.811 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:28 smithi023 ceph-mon[25869]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-24T05:30:28.811 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:28 smithi023 ceph-mon[25869]: Updating smithi023:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:30:28.811 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:28 smithi023 ceph-mon[25869]: Updating smithi063:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:30:28.811 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:28 smithi023 ceph-mon[25869]: Updating smithi143:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:30:28.811 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:28 smithi023 ceph-mon[25869]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-24T05:30:28.811 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:28 smithi023 ceph-mon[25869]: Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-03-24T05:30:28.811 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:28 smithi023 ceph-mon[25869]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-03-24T05:30:28.811 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:28 smithi023 ceph-mon[25869]: Updating smithi023:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T05:30:28.811 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:28 smithi023 ceph-mon[25869]: Updating smithi143:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T05:30:28.811 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:28 smithi023 ceph-mon[25869]: Updating smithi063:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T05:30:29.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:28 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:29.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:28 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/337553466' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T05:30:29.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:28 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:29.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:28 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T05:30:29.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:28 smithi063 ceph-mon[30595]: Adjusting osd_memory_target on smithi023 to 16064M 2024-03-24T05:30:29.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:28 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:29.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:28 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:30:29.305 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:28 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:30:29.305 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:28 smithi063 ceph-mon[30595]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-24T05:30:29.305 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:28 smithi063 ceph-mon[30595]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-24T05:30:29.305 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:28 smithi063 ceph-mon[30595]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T05:30:29.305 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:28 smithi063 ceph-mon[30595]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-24T05:30:29.305 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:28 smithi063 ceph-mon[30595]: Updating smithi023:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:30:29.305 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:28 smithi063 ceph-mon[30595]: Updating smithi063:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:30:29.305 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:28 smithi063 ceph-mon[30595]: Updating smithi143:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:30:29.305 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:28 smithi063 ceph-mon[30595]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-24T05:30:29.305 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:28 smithi063 ceph-mon[30595]: Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-03-24T05:30:29.305 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:28 smithi063 ceph-mon[30595]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-03-24T05:30:29.306 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:28 smithi063 ceph-mon[30595]: Updating smithi023:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T05:30:29.306 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:28 smithi063 ceph-mon[30595]: Updating smithi143:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T05:30:29.306 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:28 smithi063 ceph-mon[30595]: Updating smithi063:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T05:30:29.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:28 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:29.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:28 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/337553466' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T05:30:29.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:28 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:29.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:28 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T05:30:29.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:28 smithi143 ceph-mon[30288]: Adjusting osd_memory_target on smithi023 to 16064M 2024-03-24T05:30:29.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:28 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:29.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:28 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T05:30:29.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:28 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T05:30:29.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:28 smithi143 ceph-mon[30288]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-24T05:30:29.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:28 smithi143 ceph-mon[30288]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-24T05:30:29.309 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:28 smithi143 ceph-mon[30288]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T05:30:29.309 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:28 smithi143 ceph-mon[30288]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-24T05:30:29.309 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:28 smithi143 ceph-mon[30288]: Updating smithi023:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:30:29.309 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:28 smithi143 ceph-mon[30288]: Updating smithi063:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:30:29.309 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:28 smithi143 ceph-mon[30288]: Updating smithi143:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T05:30:29.309 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:28 smithi143 ceph-mon[30288]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-24T05:30:29.309 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:28 smithi143 ceph-mon[30288]: Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-03-24T05:30:29.309 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:28 smithi143 ceph-mon[30288]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-03-24T05:30:29.309 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:28 smithi143 ceph-mon[30288]: Updating smithi023:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T05:30:29.309 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:28 smithi143 ceph-mon[30288]: Updating smithi143:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T05:30:29.309 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:28 smithi143 ceph-mon[30288]: Updating smithi063:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T05:30:29.909 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:29 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:29.909 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:29 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:29.909 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:29 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:29.910 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:29 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:29.910 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:29 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:29.910 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:29 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:29.910 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:29 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:29.910 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:29 smithi023 ceph-mon[25869]: Deploying daemon alertmanager.a on smithi023 2024-03-24T05:30:30.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:29 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:30.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:29 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:30.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:29 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:30.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:29 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:30.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:29 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:30.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:29 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:30.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:29 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:30.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:29 smithi063 ceph-mon[30595]: Deploying daemon alertmanager.a on smithi023 2024-03-24T05:30:30.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:29 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:30.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:29 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:30.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:29 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:30.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:29 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:30.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:29 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:30.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:29 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:30.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:29 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:30.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:29 smithi143 ceph-mon[30288]: Deploying daemon alertmanager.a on smithi023 2024-03-24T05:30:30.613 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:30:30.615 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-24T05:30:31.239 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:30 smithi023 ceph-mon[25869]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-24T05:30:31.239 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:30 smithi023 ceph-mon[25869]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T05:30:31.266 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-03-24T05:30:29.949589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":1262,"num_read_kb":1077,"num_write":889,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":867,"ondisk_log_size":867,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96612,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126556,"statfs":{"total":287972524032,"available":287873593344,"internally_reserved":0,"allocated":3158016,"data_stored":2100351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.000988"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937329+0000","last_change":"2024-03-24T05:29:57.725554+0000","last_active":"2024-03-24T05:30:19.937329+0000","last_peered":"2024-03-24T05:30:19.937329+0000","last_clean":"2024-03-24T05:30:19.937329+0000","last_became_active":"2024-03-24T05:29:57.724358+0000","last_became_peered":"2024-03-24T05:29:57.724358+0000","last_unstale":"2024-03-24T05:30:19.937329+0000","last_undegraded":"2024-03-24T05:30:19.937329+0000","last_fullsized":"2024-03-24T05:30:19.937329+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:50:53.604623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937930+0000","last_change":"2024-03-24T05:29:55.712962+0000","last_active":"2024-03-24T05:30:19.937930+0000","last_peered":"2024-03-24T05:30:19.937930+0000","last_clean":"2024-03-24T05:30:19.937930+0000","last_became_active":"2024-03-24T05:29:55.712686+0000","last_became_peered":"2024-03-24T05:29:55.712686+0000","last_unstale":"2024-03-24T05:30:19.937930+0000","last_undegraded":"2024-03-24T05:30:19.937930+0000","last_fullsized":"2024-03-24T05:30:19.937930+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:56:30.794523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941461+0000","last_change":"2024-03-24T05:29:51.691913+0000","last_active":"2024-03-24T05:30:19.941461+0000","last_peered":"2024-03-24T05:30:19.941461+0000","last_clean":"2024-03-24T05:30:19.941461+0000","last_became_active":"2024-03-24T05:29:51.691620+0000","last_became_peered":"2024-03-24T05:29:51.691620+0000","last_unstale":"2024-03-24T05:30:19.941461+0000","last_undegraded":"2024-03-24T05:30:19.941461+0000","last_fullsized":"2024-03-24T05:30:19.941461+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:15.001586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.942027+0000","last_change":"2024-03-24T05:29:53.703805+0000","last_active":"2024-03-24T05:30:19.942027+0000","last_peered":"2024-03-24T05:30:19.942027+0000","last_clean":"2024-03-24T05:30:19.942027+0000","last_became_active":"2024-03-24T05:29:53.703408+0000","last_became_peered":"2024-03-24T05:29:53.703408+0000","last_unstale":"2024-03-24T05:30:19.942027+0000","last_undegraded":"2024-03-24T05:30:19.942027+0000","last_fullsized":"2024-03-24T05:30:19.942027+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:40:42.134534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938367+0000","last_change":"2024-03-24T05:29:57.728119+0000","last_active":"2024-03-24T05:30:19.938367+0000","last_peered":"2024-03-24T05:30:19.938367+0000","last_clean":"2024-03-24T05:30:19.938367+0000","last_became_active":"2024-03-24T05:29:57.727599+0000","last_became_peered":"2024-03-24T05:29:57.727599+0000","last_unstale":"2024-03-24T05:30:19.938367+0000","last_undegraded":"2024-03-24T05:30:19.938367+0000","last_fullsized":"2024-03-24T05:30:19.938367+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:12:19.388327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937035+0000","last_change":"2024-03-24T05:29:55.717109+0000","last_active":"2024-03-24T05:30:19.937035+0000","last_peered":"2024-03-24T05:30:19.937035+0000","last_clean":"2024-03-24T05:30:19.937035+0000","last_became_active":"2024-03-24T05:29:55.715969+0000","last_became_peered":"2024-03-24T05:29:55.715969+0000","last_unstale":"2024-03-24T05:30:19.937035+0000","last_undegraded":"2024-03-24T05:30:19.937035+0000","last_fullsized":"2024-03-24T05:30:19.937035+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:58:35.185051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.942184+0000","last_change":"2024-03-24T05:29:51.692863+0000","last_active":"2024-03-24T05:30:19.942184+0000","last_peered":"2024-03-24T05:30:19.942184+0000","last_clean":"2024-03-24T05:30:19.942184+0000","last_became_active":"2024-03-24T05:29:51.692559+0000","last_became_peered":"2024-03-24T05:29:51.692559+0000","last_unstale":"2024-03-24T05:30:19.942184+0000","last_undegraded":"2024-03-24T05:30:19.942184+0000","last_fullsized":"2024-03-24T05:30:19.942184+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:58.106532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941005+0000","last_change":"2024-03-24T05:29:53.705316+0000","last_active":"2024-03-24T05:30:19.941005+0000","last_peered":"2024-03-24T05:30:19.941005+0000","last_clean":"2024-03-24T05:30:19.941005+0000","last_became_active":"2024-03-24T05:29:53.704880+0000","last_became_peered":"2024-03-24T05:29:53.704880+0000","last_unstale":"2024-03-24T05:30:19.941005+0000","last_undegraded":"2024-03-24T05:30:19.941005+0000","last_fullsized":"2024-03-24T05:30:19.941005+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:58:35.687715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938039+0000","last_change":"2024-03-24T05:29:57.723671+0000","last_active":"2024-03-24T05:30:19.938039+0000","last_peered":"2024-03-24T05:30:19.938039+0000","last_clean":"2024-03-24T05:30:19.938039+0000","last_became_active":"2024-03-24T05:29:57.723133+0000","last_became_peered":"2024-03-24T05:29:57.723133+0000","last_unstale":"2024-03-24T05:30:19.938039+0000","last_undegraded":"2024-03-24T05:30:19.938039+0000","last_fullsized":"2024-03-24T05:30:19.938039+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:38:56.045037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937550+0000","last_change":"2024-03-24T05:29:55.717322+0000","last_active":"2024-03-24T05:30:19.937550+0000","last_peered":"2024-03-24T05:30:19.937550+0000","last_clean":"2024-03-24T05:30:19.937550+0000","last_became_active":"2024-03-24T05:29:55.716932+0000","last_became_peered":"2024-03-24T05:29:55.716932+0000","last_unstale":"2024-03-24T05:30:19.937550+0000","last_undegraded":"2024-03-24T05:30:19.937550+0000","last_fullsized":"2024-03-24T05:30:19.937550+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:38:53.342699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941119+0000","last_change":"2024-03-24T05:29:51.690829+0000","last_active":"2024-03-24T05:30:19.941119+0000","last_peered":"2024-03-24T05:30:19.941119+0000","last_clean":"2024-03-24T05:30:19.941119+0000","last_became_active":"2024-03-24T05:29:51.688559+0000","last_became_peered":"2024-03-24T05:29:51.688559+0000","last_unstale":"2024-03-24T05:30:19.941119+0000","last_undegraded":"2024-03-24T05:30:19.941119+0000","last_fullsized":"2024-03-24T05:30:19.941119+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:03:33.365876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939106+0000","last_change":"2024-03-24T05:29:53.697105+0000","last_active":"2024-03-24T05:30:19.939106+0000","last_peered":"2024-03-24T05:30:19.939106+0000","last_clean":"2024-03-24T05:30:19.939106+0000","last_became_active":"2024-03-24T05:29:53.696742+0000","last_became_peered":"2024-03-24T05:29:53.696742+0000","last_unstale":"2024-03-24T05:30:19.939106+0000","last_undegraded":"2024-03-24T05:30:19.939106+0000","last_fullsized":"2024-03-24T05:30:19.939106+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:21:08.421214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937443+0000","last_change":"2024-03-24T05:29:57.725621+0000","last_active":"2024-03-24T05:30:19.937443+0000","last_peered":"2024-03-24T05:30:19.937443+0000","last_clean":"2024-03-24T05:30:19.937443+0000","last_became_active":"2024-03-24T05:29:57.723737+0000","last_became_peered":"2024-03-24T05:29:57.723737+0000","last_unstale":"2024-03-24T05:30:19.937443+0000","last_undegraded":"2024-03-24T05:30:19.937443+0000","last_fullsized":"2024-03-24T05:30:19.937443+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:37:20.857940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937131+0000","last_change":"2024-03-24T05:29:55.715512+0000","last_active":"2024-03-24T05:30:19.937131+0000","last_peered":"2024-03-24T05:30:19.937131+0000","last_clean":"2024-03-24T05:30:19.937131+0000","last_became_active":"2024-03-24T05:29:55.714880+0000","last_became_peered":"2024-03-24T05:29:55.714880+0000","last_unstale":"2024-03-24T05:30:19.937131+0000","last_undegraded":"2024-03-24T05:30:19.937131+0000","last_fullsized":"2024-03-24T05:30:19.937131+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:02.629129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939316+0000","last_change":"2024-03-24T05:29:51.688533+0000","last_active":"2024-03-24T05:30:19.939316+0000","last_peered":"2024-03-24T05:30:19.939316+0000","last_clean":"2024-03-24T05:30:19.939316+0000","last_became_active":"2024-03-24T05:29:51.687764+0000","last_became_peered":"2024-03-24T05:29:51.687764+0000","last_unstale":"2024-03-24T05:30:19.939316+0000","last_undegraded":"2024-03-24T05:30:19.939316+0000","last_fullsized":"2024-03-24T05:30:19.939316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:02:51.403460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'18","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941066+0000","last_change":"2024-03-24T05:29:53.698767+0000","last_active":"2024-03-24T05:30:19.941066+0000","last_peered":"2024-03-24T05:30:19.941066+0000","last_clean":"2024-03-24T05:30:19.941066+0000","last_became_active":"2024-03-24T05:29:53.698257+0000","last_became_peered":"2024-03-24T05:29:53.698257+0000","last_unstale":"2024-03-24T05:30:19.941066+0000","last_undegraded":"2024-03-24T05:30:19.941066+0000","last_fullsized":"2024-03-24T05:30:19.941066+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:45:55.798554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938396+0000","last_change":"2024-03-24T05:29:57.725054+0000","last_active":"2024-03-24T05:30:19.938396+0000","last_peered":"2024-03-24T05:30:19.938396+0000","last_clean":"2024-03-24T05:30:19.938396+0000","last_became_active":"2024-03-24T05:29:57.723935+0000","last_became_peered":"2024-03-24T05:29:57.723935+0000","last_unstale":"2024-03-24T05:30:19.938396+0000","last_undegraded":"2024-03-24T05:30:19.938396+0000","last_fullsized":"2024-03-24T05:30:19.938396+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:45:20.040217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939946+0000","last_change":"2024-03-24T05:29:55.716731+0000","last_active":"2024-03-24T05:30:19.939946+0000","last_peered":"2024-03-24T05:30:19.939946+0000","last_clean":"2024-03-24T05:30:19.939946+0000","last_became_active":"2024-03-24T05:29:55.716303+0000","last_became_peered":"2024-03-24T05:29:55.716303+0000","last_unstale":"2024-03-24T05:30:19.939946+0000","last_undegraded":"2024-03-24T05:30:19.939946+0000","last_fullsized":"2024-03-24T05:30:19.939946+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:46:09.397749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939409+0000","last_change":"2024-03-24T05:29:51.688388+0000","last_active":"2024-03-24T05:30:19.939409+0000","last_peered":"2024-03-24T05:30:19.939409+0000","last_clean":"2024-03-24T05:30:19.939409+0000","last_became_active":"2024-03-24T05:29:51.686710+0000","last_became_peered":"2024-03-24T05:29:51.686710+0000","last_unstale":"2024-03-24T05:30:19.939409+0000","last_undegraded":"2024-03-24T05:30:19.939409+0000","last_fullsized":"2024-03-24T05:30:19.939409+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:14.513640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'21","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938806+0000","last_change":"2024-03-24T05:29:53.700518+0000","last_active":"2024-03-24T05:30:19.938806+0000","last_peered":"2024-03-24T05:30:19.938806+0000","last_clean":"2024-03-24T05:30:19.938806+0000","last_became_active":"2024-03-24T05:29:53.700218+0000","last_became_peered":"2024-03-24T05:29:53.700218+0000","last_unstale":"2024-03-24T05:30:19.938806+0000","last_undegraded":"2024-03-24T05:30:19.938806+0000","last_fullsized":"2024-03-24T05:30:19.938806+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:12:06.439923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939906+0000","last_change":"2024-03-24T05:29:57.729024+0000","last_active":"2024-03-24T05:30:19.939906+0000","last_peered":"2024-03-24T05:30:19.939906+0000","last_clean":"2024-03-24T05:30:19.939906+0000","last_became_active":"2024-03-24T05:29:57.728713+0000","last_became_peered":"2024-03-24T05:29:57.728713+0000","last_unstale":"2024-03-24T05:30:19.939906+0000","last_undegraded":"2024-03-24T05:30:19.939906+0000","last_fullsized":"2024-03-24T05:30:19.939906+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:21.817744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939082+0000","last_change":"2024-03-24T05:29:55.713007+0000","last_active":"2024-03-24T05:30:19.939082+0000","last_peered":"2024-03-24T05:30:19.939082+0000","last_clean":"2024-03-24T05:30:19.939082+0000","last_became_active":"2024-03-24T05:29:55.712701+0000","last_became_peered":"2024-03-24T05:29:55.712701+0000","last_unstale":"2024-03-24T05:30:19.939082+0000","last_undegraded":"2024-03-24T05:30:19.939082+0000","last_fullsized":"2024-03-24T05:30:19.939082+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:44:23.593104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937360+0000","last_change":"2024-03-24T05:29:51.687604+0000","last_active":"2024-03-24T05:30:19.937360+0000","last_peered":"2024-03-24T05:30:19.937360+0000","last_clean":"2024-03-24T05:30:19.937360+0000","last_became_active":"2024-03-24T05:29:51.687205+0000","last_became_peered":"2024-03-24T05:29:51.687205+0000","last_unstale":"2024-03-24T05:30:19.937360+0000","last_undegraded":"2024-03-24T05:30:19.937360+0000","last_fullsized":"2024-03-24T05:30:19.937360+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:05:04.180724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'11","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939055+0000","last_change":"2024-03-24T05:29:53.700618+0000","last_active":"2024-03-24T05:30:19.939055+0000","last_peered":"2024-03-24T05:30:19.939055+0000","last_clean":"2024-03-24T05:30:19.939055+0000","last_became_active":"2024-03-24T05:29:53.700243+0000","last_became_peered":"2024-03-24T05:29:53.700243+0000","last_unstale":"2024-03-24T05:30:19.939055+0000","last_undegraded":"2024-03-24T05:30:19.939055+0000","last_fullsized":"2024-03-24T05:30:19.939055+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:03:29.518774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939201+0000","last_change":"2024-03-24T05:29:57.728603+0000","last_active":"2024-03-24T05:30:19.939201+0000","last_peered":"2024-03-24T05:30:19.939201+0000","last_clean":"2024-03-24T05:30:19.939201+0000","last_became_active":"2024-03-24T05:29:57.728097+0000","last_became_peered":"2024-03-24T05:29:57.728097+0000","last_unstale":"2024-03-24T05:30:19.939201+0000","last_undegraded":"2024-03-24T05:30:19.939201+0000","last_fullsized":"2024-03-24T05:30:19.939201+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:52.482520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939522+0000","last_change":"2024-03-24T05:29:55.717112+0000","last_active":"2024-03-24T05:30:19.939522+0000","last_peered":"2024-03-24T05:30:19.939522+0000","last_clean":"2024-03-24T05:30:19.939522+0000","last_became_active":"2024-03-24T05:29:55.715949+0000","last_became_peered":"2024-03-24T05:29:55.715949+0000","last_unstale":"2024-03-24T05:30:19.939522+0000","last_undegraded":"2024-03-24T05:30:19.939522+0000","last_fullsized":"2024-03-24T05:30:19.939522+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:53:05.607989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938724+0000","last_change":"2024-03-24T05:29:51.691411+0000","last_active":"2024-03-24T05:30:19.938724+0000","last_peered":"2024-03-24T05:30:19.938724+0000","last_clean":"2024-03-24T05:30:19.938724+0000","last_became_active":"2024-03-24T05:29:51.691023+0000","last_became_peered":"2024-03-24T05:29:51.691023+0000","last_unstale":"2024-03-24T05:30:19.938724+0000","last_undegraded":"2024-03-24T05:30:19.938724+0000","last_fullsized":"2024-03-24T05:30:19.938724+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:04:48.345391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937742+0000","last_change":"2024-03-24T05:29:53.702446+0000","last_active":"2024-03-24T05:30:19.937742+0000","last_peered":"2024-03-24T05:30:19.937742+0000","last_clean":"2024-03-24T05:30:19.937742+0000","last_became_active":"2024-03-24T05:29:53.702144+0000","last_became_peered":"2024-03-24T05:29:53.702144+0000","last_unstale":"2024-03-24T05:30:19.937742+0000","last_undegraded":"2024-03-24T05:30:19.937742+0000","last_fullsized":"2024-03-24T05:30:19.937742+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:59:30.591850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938656+0000","last_change":"2024-03-24T05:29:57.726574+0000","last_active":"2024-03-24T05:30:19.938656+0000","last_peered":"2024-03-24T05:30:19.938656+0000","last_clean":"2024-03-24T05:30:19.938656+0000","last_became_active":"2024-03-24T05:29:57.724489+0000","last_became_peered":"2024-03-24T05:29:57.724489+0000","last_unstale":"2024-03-24T05:30:19.938656+0000","last_undegraded":"2024-03-24T05:30:19.938656+0000","last_fullsized":"2024-03-24T05:30:19.938656+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:18.385378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941700+0000","last_change":"2024-03-24T05:29:55.715847+0000","last_active":"2024-03-24T05:30:19.941700+0000","last_peered":"2024-03-24T05:30:19.941700+0000","last_clean":"2024-03-24T05:30:19.941700+0000","last_became_active":"2024-03-24T05:29:55.715497+0000","last_became_peered":"2024-03-24T05:29:55.715497+0000","last_unstale":"2024-03-24T05:30:19.941700+0000","last_undegraded":"2024-03-24T05:30:19.941700+0000","last_fullsized":"2024-03-24T05:30:19.941700+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:12:37.665790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938380+0000","last_change":"2024-03-24T05:29:51.691384+0000","last_active":"2024-03-24T05:30:19.938380+0000","last_peered":"2024-03-24T05:30:19.938380+0000","last_clean":"2024-03-24T05:30:19.938380+0000","last_became_active":"2024-03-24T05:29:51.689458+0000","last_became_peered":"2024-03-24T05:29:51.689458+0000","last_unstale":"2024-03-24T05:30:19.938380+0000","last_undegraded":"2024-03-24T05:30:19.938380+0000","last_fullsized":"2024-03-24T05:30:19.938380+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:52:03.687301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'39","reported_seq":96,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937967+0000","last_change":"2024-03-24T05:29:53.702302+0000","last_active":"2024-03-24T05:30:19.937967+0000","last_peered":"2024-03-24T05:30:19.937967+0000","last_clean":"2024-03-24T05:30:19.937967+0000","last_became_active":"2024-03-24T05:29:53.701917+0000","last_became_peered":"2024-03-24T05:29:53.701917+0000","last_unstale":"2024-03-24T05:30:19.937967+0000","last_undegraded":"2024-03-24T05:30:19.937967+0000","last_fullsized":"2024-03-24T05:30:19.937967+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:31.597068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937217+0000","last_change":"2024-03-24T05:29:57.723600+0000","last_active":"2024-03-24T05:30:19.937217+0000","last_peered":"2024-03-24T05:30:19.937217+0000","last_clean":"2024-03-24T05:30:19.937217+0000","last_became_active":"2024-03-24T05:29:57.722672+0000","last_became_peered":"2024-03-24T05:29:57.722672+0000","last_unstale":"2024-03-24T05:30:19.937217+0000","last_undegraded":"2024-03-24T05:30:19.937217+0000","last_fullsized":"2024-03-24T05:30:19.937217+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:05:37.886835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937827+0000","last_change":"2024-03-24T05:29:55.712734+0000","last_active":"2024-03-24T05:30:19.937827+0000","last_peered":"2024-03-24T05:30:19.937827+0000","last_clean":"2024-03-24T05:30:19.937827+0000","last_became_active":"2024-03-24T05:29:55.712361+0000","last_became_peered":"2024-03-24T05:29:55.712361+0000","last_unstale":"2024-03-24T05:30:19.937827+0000","last_undegraded":"2024-03-24T05:30:19.937827+0000","last_fullsized":"2024-03-24T05:30:19.937827+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:55:28.982750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939645+0000","last_change":"2024-03-24T05:29:51.687454+0000","last_active":"2024-03-24T05:30:19.939645+0000","last_peered":"2024-03-24T05:30:19.939645+0000","last_clean":"2024-03-24T05:30:19.939645+0000","last_became_active":"2024-03-24T05:29:51.686318+0000","last_became_peered":"2024-03-24T05:29:51.686318+0000","last_unstale":"2024-03-24T05:30:19.939645+0000","last_undegraded":"2024-03-24T05:30:19.939645+0000","last_fullsized":"2024-03-24T05:30:19.939645+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:55:43.706601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'21","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939175+0000","last_change":"2024-03-24T05:29:53.698120+0000","last_active":"2024-03-24T05:30:19.939175+0000","last_peered":"2024-03-24T05:30:19.939175+0000","last_clean":"2024-03-24T05:30:19.939175+0000","last_became_active":"2024-03-24T05:29:53.697313+0000","last_became_peered":"2024-03-24T05:29:53.697313+0000","last_unstale":"2024-03-24T05:30:19.939175+0000","last_undegraded":"2024-03-24T05:30:19.939175+0000","last_fullsized":"2024-03-24T05:30:19.939175+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:37.043805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937693+0000","last_change":"2024-03-24T05:29:57.724474+0000","last_active":"2024-03-24T05:30:19.937693+0000","last_peered":"2024-03-24T05:30:19.937693+0000","last_clean":"2024-03-24T05:30:19.937693+0000","last_became_active":"2024-03-24T05:29:57.723522+0000","last_became_peered":"2024-03-24T05:29:57.723522+0000","last_unstale":"2024-03-24T05:30:19.937693+0000","last_undegraded":"2024-03-24T05:30:19.937693+0000","last_fullsized":"2024-03-24T05:30:19.937693+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:58.222918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938232+0000","last_change":"2024-03-24T05:29:55.716186+0000","last_active":"2024-03-24T05:30:19.938232+0000","last_peered":"2024-03-24T05:30:19.938232+0000","last_clean":"2024-03-24T05:30:19.938232+0000","last_became_active":"2024-03-24T05:29:55.715910+0000","last_became_peered":"2024-03-24T05:29:55.715910+0000","last_unstale":"2024-03-24T05:30:19.938232+0000","last_undegraded":"2024-03-24T05:30:19.938232+0000","last_fullsized":"2024-03-24T05:30:19.938232+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:02:06.223913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938741+0000","last_change":"2024-03-24T05:29:51.685304+0000","last_active":"2024-03-24T05:30:19.938741+0000","last_peered":"2024-03-24T05:30:19.938741+0000","last_clean":"2024-03-24T05:30:19.938741+0000","last_became_active":"2024-03-24T05:29:51.684613+0000","last_became_peered":"2024-03-24T05:29:51.684613+0000","last_unstale":"2024-03-24T05:30:19.938741+0000","last_undegraded":"2024-03-24T05:30:19.938741+0000","last_fullsized":"2024-03-24T05:30:19.938741+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:24.326969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'12","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941225+0000","last_change":"2024-03-24T05:29:53.703499+0000","last_active":"2024-03-24T05:30:19.941225+0000","last_peered":"2024-03-24T05:30:19.941225+0000","last_clean":"2024-03-24T05:30:19.941225+0000","last_became_active":"2024-03-24T05:29:53.702938+0000","last_became_peered":"2024-03-24T05:29:53.702938+0000","last_unstale":"2024-03-24T05:30:19.941225+0000","last_undegraded":"2024-03-24T05:30:19.941225+0000","last_fullsized":"2024-03-24T05:30:19.941225+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:05.424232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938202+0000","last_change":"2024-03-24T05:29:57.723093+0000","last_active":"2024-03-24T05:30:19.938202+0000","last_peered":"2024-03-24T05:30:19.938202+0000","last_clean":"2024-03-24T05:30:19.938202+0000","last_became_active":"2024-03-24T05:29:57.722791+0000","last_became_peered":"2024-03-24T05:29:57.722791+0000","last_unstale":"2024-03-24T05:30:19.938202+0000","last_undegraded":"2024-03-24T05:30:19.938202+0000","last_fullsized":"2024-03-24T05:30:19.938202+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:55:37.633776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937808+0000","last_change":"2024-03-24T05:29:55.717043+0000","last_active":"2024-03-24T05:30:19.937808+0000","last_peered":"2024-03-24T05:30:19.937808+0000","last_clean":"2024-03-24T05:30:19.937808+0000","last_became_active":"2024-03-24T05:29:55.715968+0000","last_became_peered":"2024-03-24T05:29:55.715968+0000","last_unstale":"2024-03-24T05:30:19.937808+0000","last_undegraded":"2024-03-24T05:30:19.937808+0000","last_fullsized":"2024-03-24T05:30:19.937808+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:29:40.714692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941013+0000","last_change":"2024-03-24T05:29:51.690997+0000","last_active":"2024-03-24T05:30:19.941013+0000","last_peered":"2024-03-24T05:30:19.941013+0000","last_clean":"2024-03-24T05:30:19.941013+0000","last_became_active":"2024-03-24T05:29:51.689504+0000","last_became_peered":"2024-03-24T05:29:51.689504+0000","last_unstale":"2024-03-24T05:30:19.941013+0000","last_undegraded":"2024-03-24T05:30:19.941013+0000","last_fullsized":"2024-03-24T05:30:19.941013+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:42.338022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'22","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939380+0000","last_change":"2024-03-24T05:29:53.698083+0000","last_active":"2024-03-24T05:30:19.939380+0000","last_peered":"2024-03-24T05:30:19.939380+0000","last_clean":"2024-03-24T05:30:19.939380+0000","last_became_active":"2024-03-24T05:29:53.697763+0000","last_became_peered":"2024-03-24T05:29:53.697763+0000","last_unstale":"2024-03-24T05:30:19.939380+0000","last_undegraded":"2024-03-24T05:30:19.939380+0000","last_fullsized":"2024-03-24T05:30:19.939380+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:47:29.474207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938199+0000","last_change":"2024-03-24T05:29:57.723770+0000","last_active":"2024-03-24T05:30:19.938199+0000","last_peered":"2024-03-24T05:30:19.938199+0000","last_clean":"2024-03-24T05:30:19.938199+0000","last_became_active":"2024-03-24T05:29:57.723278+0000","last_became_peered":"2024-03-24T05:29:57.723278+0000","last_unstale":"2024-03-24T05:30:19.938199+0000","last_undegraded":"2024-03-24T05:30:19.938199+0000","last_fullsized":"2024-03-24T05:30:19.938199+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:03:58.258463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938077+0000","last_change":"2024-03-24T05:29:55.715161+0000","last_active":"2024-03-24T05:30:19.938077+0000","last_peered":"2024-03-24T05:30:19.938077+0000","last_clean":"2024-03-24T05:30:19.938077+0000","last_became_active":"2024-03-24T05:29:55.714722+0000","last_became_peered":"2024-03-24T05:29:55.714722+0000","last_unstale":"2024-03-24T05:30:19.938077+0000","last_undegraded":"2024-03-24T05:30:19.938077+0000","last_fullsized":"2024-03-24T05:30:19.938077+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:54:34.389924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938865+0000","last_change":"2024-03-24T05:29:51.688513+0000","last_active":"2024-03-24T05:30:19.938865+0000","last_peered":"2024-03-24T05:30:19.938865+0000","last_clean":"2024-03-24T05:30:19.938865+0000","last_became_active":"2024-03-24T05:29:51.686135+0000","last_became_peered":"2024-03-24T05:29:51.686135+0000","last_unstale":"2024-03-24T05:30:19.938865+0000","last_undegraded":"2024-03-24T05:30:19.938865+0000","last_fullsized":"2024-03-24T05:30:19.938865+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:09:06.634054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940176+0000","last_change":"2024-03-24T05:29:53.698168+0000","last_active":"2024-03-24T05:30:19.940176+0000","last_peered":"2024-03-24T05:30:19.940176+0000","last_clean":"2024-03-24T05:30:19.940176+0000","last_became_active":"2024-03-24T05:29:53.697881+0000","last_became_peered":"2024-03-24T05:29:53.697881+0000","last_unstale":"2024-03-24T05:30:19.940176+0000","last_undegraded":"2024-03-24T05:30:19.940176+0000","last_fullsized":"2024-03-24T05:30:19.940176+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:29:49.365852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940134+0000","last_change":"2024-03-24T05:29:57.728360+0000","last_active":"2024-03-24T05:30:19.940134+0000","last_peered":"2024-03-24T05:30:19.940134+0000","last_clean":"2024-03-24T05:30:19.940134+0000","last_became_active":"2024-03-24T05:29:57.728052+0000","last_became_peered":"2024-03-24T05:29:57.728052+0000","last_unstale":"2024-03-24T05:30:19.940134+0000","last_undegraded":"2024-03-24T05:30:19.940134+0000","last_fullsized":"2024-03-24T05:30:19.940134+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:13:15.314499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940097+0000","last_change":"2024-03-24T05:29:55.716925+0000","last_active":"2024-03-24T05:30:19.940097+0000","last_peered":"2024-03-24T05:30:19.940097+0000","last_clean":"2024-03-24T05:30:19.940097+0000","last_became_active":"2024-03-24T05:29:55.716516+0000","last_became_peered":"2024-03-24T05:29:55.716516+0000","last_unstale":"2024-03-24T05:30:19.940097+0000","last_undegraded":"2024-03-24T05:30:19.940097+0000","last_fullsized":"2024-03-24T05:30:19.940097+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:39.074860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938832+0000","last_change":"2024-03-24T05:29:51.690453+0000","last_active":"2024-03-24T05:30:19.938832+0000","last_peered":"2024-03-24T05:30:19.938832+0000","last_clean":"2024-03-24T05:30:19.938832+0000","last_became_active":"2024-03-24T05:29:51.689378+0000","last_became_peered":"2024-03-24T05:29:51.689378+0000","last_unstale":"2024-03-24T05:30:19.938832+0000","last_undegraded":"2024-03-24T05:30:19.938832+0000","last_fullsized":"2024-03-24T05:30:19.938832+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:35:50.258253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938225+0000","last_change":"2024-03-24T05:29:53.702889+0000","last_active":"2024-03-24T05:30:19.938225+0000","last_peered":"2024-03-24T05:30:19.938225+0000","last_clean":"2024-03-24T05:30:19.938225+0000","last_became_active":"2024-03-24T05:29:53.702386+0000","last_became_peered":"2024-03-24T05:29:53.702386+0000","last_unstale":"2024-03-24T05:30:19.938225+0000","last_undegraded":"2024-03-24T05:30:19.938225+0000","last_fullsized":"2024-03-24T05:30:19.938225+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:13.514023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938024+0000","last_change":"2024-03-24T05:29:57.726006+0000","last_active":"2024-03-24T05:30:19.938024+0000","last_peered":"2024-03-24T05:30:19.938024+0000","last_clean":"2024-03-24T05:30:19.938024+0000","last_became_active":"2024-03-24T05:29:57.725645+0000","last_became_peered":"2024-03-24T05:29:57.725645+0000","last_unstale":"2024-03-24T05:30:19.938024+0000","last_undegraded":"2024-03-24T05:30:19.938024+0000","last_fullsized":"2024-03-24T05:30:19.938024+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:36:01.547224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939580+0000","last_change":"2024-03-24T05:29:55.712011+0000","last_active":"2024-03-24T05:30:19.939580+0000","last_peered":"2024-03-24T05:30:19.939580+0000","last_clean":"2024-03-24T05:30:19.939580+0000","last_became_active":"2024-03-24T05:29:55.711716+0000","last_became_peered":"2024-03-24T05:29:55.711716+0000","last_unstale":"2024-03-24T05:30:19.939580+0000","last_undegraded":"2024-03-24T05:30:19.939580+0000","last_fullsized":"2024-03-24T05:30:19.939580+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:31:18.147946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938331+0000","last_change":"2024-03-24T05:29:51.690262+0000","last_active":"2024-03-24T05:30:19.938331+0000","last_peered":"2024-03-24T05:30:19.938331+0000","last_clean":"2024-03-24T05:30:19.938331+0000","last_became_active":"2024-03-24T05:29:51.687739+0000","last_became_peered":"2024-03-24T05:29:51.687739+0000","last_unstale":"2024-03-24T05:30:19.938331+0000","last_undegraded":"2024-03-24T05:30:19.938331+0000","last_fullsized":"2024-03-24T05:30:19.938331+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:47.683861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938088+0000","last_change":"2024-03-24T05:29:53.697826+0000","last_active":"2024-03-24T05:30:19.938088+0000","last_peered":"2024-03-24T05:30:19.938088+0000","last_clean":"2024-03-24T05:30:19.938088+0000","last_became_active":"2024-03-24T05:29:53.697636+0000","last_became_peered":"2024-03-24T05:29:53.697636+0000","last_unstale":"2024-03-24T05:30:19.938088+0000","last_undegraded":"2024-03-24T05:30:19.938088+0000","last_fullsized":"2024-03-24T05:30:19.938088+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:38:19.391690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938204+0000","last_change":"2024-03-24T05:29:57.726588+0000","last_active":"2024-03-24T05:30:19.938204+0000","last_peered":"2024-03-24T05:30:19.938204+0000","last_clean":"2024-03-24T05:30:19.938204+0000","last_became_active":"2024-03-24T05:29:57.725791+0000","last_became_peered":"2024-03-24T05:29:57.725791+0000","last_unstale":"2024-03-24T05:30:19.938204+0000","last_undegraded":"2024-03-24T05:30:19.938204+0000","last_fullsized":"2024-03-24T05:30:19.938204+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:51:29.405480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940821+0000","last_change":"2024-03-24T05:29:55.719395+0000","last_active":"2024-03-24T05:30:19.940821+0000","last_peered":"2024-03-24T05:30:19.940821+0000","last_clean":"2024-03-24T05:30:19.940821+0000","last_became_active":"2024-03-24T05:29:55.718514+0000","last_became_peered":"2024-03-24T05:29:55.718514+0000","last_unstale":"2024-03-24T05:30:19.940821+0000","last_undegraded":"2024-03-24T05:30:19.940821+0000","last_fullsized":"2024-03-24T05:30:19.940821+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:10.861395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937064+0000","last_change":"2024-03-24T05:29:51.685235+0000","last_active":"2024-03-24T05:30:19.937064+0000","last_peered":"2024-03-24T05:30:19.937064+0000","last_clean":"2024-03-24T05:30:19.937064+0000","last_became_active":"2024-03-24T05:29:51.684606+0000","last_became_peered":"2024-03-24T05:29:51.684606+0000","last_unstale":"2024-03-24T05:30:19.937064+0000","last_undegraded":"2024-03-24T05:30:19.937064+0000","last_fullsized":"2024-03-24T05:30:19.937064+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:06:10.422744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.936758+0000","last_change":"2024-03-24T05:29:53.697197+0000","last_active":"2024-03-24T05:30:19.936758+0000","last_peered":"2024-03-24T05:30:19.936758+0000","last_clean":"2024-03-24T05:30:19.936758+0000","last_became_active":"2024-03-24T05:29:53.696835+0000","last_became_peered":"2024-03-24T05:29:53.696835+0000","last_unstale":"2024-03-24T05:30:19.936758+0000","last_undegraded":"2024-03-24T05:30:19.936758+0000","last_fullsized":"2024-03-24T05:30:19.936758+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:17:13.302550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940536+0000","last_change":"2024-03-24T05:29:57.729138+0000","last_active":"2024-03-24T05:30:19.940536+0000","last_peered":"2024-03-24T05:30:19.940536+0000","last_clean":"2024-03-24T05:30:19.940536+0000","last_became_active":"2024-03-24T05:29:57.728830+0000","last_became_peered":"2024-03-24T05:29:57.728830+0000","last_unstale":"2024-03-24T05:30:19.940536+0000","last_undegraded":"2024-03-24T05:30:19.940536+0000","last_fullsized":"2024-03-24T05:30:19.940536+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:42:16.818621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941792+0000","last_change":"2024-03-24T05:29:55.715995+0000","last_active":"2024-03-24T05:30:19.941792+0000","last_peered":"2024-03-24T05:30:19.941792+0000","last_clean":"2024-03-24T05:30:19.941792+0000","last_became_active":"2024-03-24T05:29:55.715673+0000","last_became_peered":"2024-03-24T05:29:55.715673+0000","last_unstale":"2024-03-24T05:30:19.941792+0000","last_undegraded":"2024-03-24T05:30:19.941792+0000","last_fullsized":"2024-03-24T05:30:19.941792+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:08:13.673922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938548+0000","last_change":"2024-03-24T05:29:51.690930+0000","last_active":"2024-03-24T05:30:19.938548+0000","last_peered":"2024-03-24T05:30:19.938548+0000","last_clean":"2024-03-24T05:30:19.938548+0000","last_became_active":"2024-03-24T05:29:51.688169+0000","last_became_peered":"2024-03-24T05:29:51.688169+0000","last_unstale":"2024-03-24T05:30:19.938548+0000","last_undegraded":"2024-03-24T05:30:19.938548+0000","last_fullsized":"2024-03-24T05:30:19.938548+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:12:16.953006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939501+0000","last_change":"2024-03-24T05:29:53.697410+0000","last_active":"2024-03-24T05:30:19.939501+0000","last_peered":"2024-03-24T05:30:19.939501+0000","last_clean":"2024-03-24T05:30:19.939501+0000","last_became_active":"2024-03-24T05:29:53.696603+0000","last_became_peered":"2024-03-24T05:29:53.696603+0000","last_unstale":"2024-03-24T05:30:19.939501+0000","last_undegraded":"2024-03-24T05:30:19.939501+0000","last_fullsized":"2024-03-24T05:30:19.939501+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:15:23.001096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939212+0000","last_change":"2024-03-24T05:29:57.722171+0000","last_active":"2024-03-24T05:30:19.939212+0000","last_peered":"2024-03-24T05:30:19.939212+0000","last_clean":"2024-03-24T05:30:19.939212+0000","last_became_active":"2024-03-24T05:29:57.721917+0000","last_became_peered":"2024-03-24T05:29:57.721917+0000","last_unstale":"2024-03-24T05:30:19.939212+0000","last_undegraded":"2024-03-24T05:30:19.939212+0000","last_fullsized":"2024-03-24T05:30:19.939212+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:27:37.865157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939778+0000","last_change":"2024-03-24T05:29:51.688269+0000","last_active":"2024-03-24T05:30:19.939778+0000","last_peered":"2024-03-24T05:30:19.939778+0000","last_clean":"2024-03-24T05:30:19.939778+0000","last_became_active":"2024-03-24T05:29:51.685514+0000","last_became_peered":"2024-03-24T05:29:51.685514+0000","last_unstale":"2024-03-24T05:30:19.939778+0000","last_undegraded":"2024-03-24T05:30:19.939778+0000","last_fullsized":"2024-03-24T05:30:19.939778+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:01.333873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940994+0000","last_change":"2024-03-24T05:29:53.700911+0000","last_active":"2024-03-24T05:30:19.940994+0000","last_peered":"2024-03-24T05:30:19.940994+0000","last_clean":"2024-03-24T05:30:19.940994+0000","last_became_active":"2024-03-24T05:29:53.700610+0000","last_became_peered":"2024-03-24T05:29:53.700610+0000","last_unstale":"2024-03-24T05:30:19.940994+0000","last_undegraded":"2024-03-24T05:30:19.940994+0000","last_fullsized":"2024-03-24T05:30:19.940994+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:43:18.485486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938086+0000","last_change":"2024-03-24T05:29:55.716102+0000","last_active":"2024-03-24T05:30:19.938086+0000","last_peered":"2024-03-24T05:30:19.938086+0000","last_clean":"2024-03-24T05:30:19.938086+0000","last_became_active":"2024-03-24T05:29:55.715754+0000","last_became_peered":"2024-03-24T05:29:55.715754+0000","last_unstale":"2024-03-24T05:30:19.938086+0000","last_undegraded":"2024-03-24T05:30:19.938086+0000","last_fullsized":"2024-03-24T05:30:19.938086+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:02.819322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938332+0000","last_change":"2024-03-24T05:29:57.726672+0000","last_active":"2024-03-24T05:30:19.938332+0000","last_peered":"2024-03-24T05:30:19.938332+0000","last_clean":"2024-03-24T05:30:19.938332+0000","last_became_active":"2024-03-24T05:29:57.726320+0000","last_became_peered":"2024-03-24T05:29:57.726320+0000","last_unstale":"2024-03-24T05:30:19.938332+0000","last_undegraded":"2024-03-24T05:30:19.938332+0000","last_fullsized":"2024-03-24T05:30:19.938332+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:14:33.520133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940723+0000","last_change":"2024-03-24T05:29:51.691807+0000","last_active":"2024-03-24T05:30:19.940723+0000","last_peered":"2024-03-24T05:30:19.940723+0000","last_clean":"2024-03-24T05:30:19.940723+0000","last_became_active":"2024-03-24T05:29:51.691273+0000","last_became_peered":"2024-03-24T05:29:51.691273+0000","last_unstale":"2024-03-24T05:30:19.940723+0000","last_undegraded":"2024-03-24T05:30:19.940723+0000","last_fullsized":"2024-03-24T05:30:19.940723+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:51.744600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939299+0000","last_change":"2024-03-24T05:29:53.697131+0000","last_active":"2024-03-24T05:30:19.939299+0000","last_peered":"2024-03-24T05:30:19.939299+0000","last_clean":"2024-03-24T05:30:19.939299+0000","last_became_active":"2024-03-24T05:29:53.696772+0000","last_became_peered":"2024-03-24T05:29:53.696772+0000","last_unstale":"2024-03-24T05:30:19.939299+0000","last_undegraded":"2024-03-24T05:30:19.939299+0000","last_fullsized":"2024-03-24T05:30:19.939299+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:01:16.901253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938203+0000","last_change":"2024-03-24T05:29:55.716629+0000","last_active":"2024-03-24T05:30:19.938203+0000","last_peered":"2024-03-24T05:30:19.938203+0000","last_clean":"2024-03-24T05:30:19.938203+0000","last_became_active":"2024-03-24T05:29:55.716266+0000","last_became_peered":"2024-03-24T05:29:55.716266+0000","last_unstale":"2024-03-24T05:30:19.938203+0000","last_undegraded":"2024-03-24T05:30:19.938203+0000","last_fullsized":"2024-03-24T05:30:19.938203+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:24:59.772747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939389+0000","last_change":"2024-03-24T05:29:57.727150+0000","last_active":"2024-03-24T05:30:19.939389+0000","last_peered":"2024-03-24T05:30:19.939389+0000","last_clean":"2024-03-24T05:30:19.939389+0000","last_became_active":"2024-03-24T05:29:57.726853+0000","last_became_peered":"2024-03-24T05:29:57.726853+0000","last_unstale":"2024-03-24T05:30:19.939389+0000","last_undegraded":"2024-03-24T05:30:19.939389+0000","last_fullsized":"2024-03-24T05:30:19.939389+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:30:44.744538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937459+0000","last_change":"2024-03-24T05:29:51.687266+0000","last_active":"2024-03-24T05:30:19.937459+0000","last_peered":"2024-03-24T05:30:19.937459+0000","last_clean":"2024-03-24T05:30:19.937459+0000","last_became_active":"2024-03-24T05:29:51.686218+0000","last_became_peered":"2024-03-24T05:29:51.686218+0000","last_unstale":"2024-03-24T05:30:19.937459+0000","last_undegraded":"2024-03-24T05:30:19.937459+0000","last_fullsized":"2024-03-24T05:30:19.937459+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:47:06.136824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'25","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938289+0000","last_change":"2024-03-24T05:29:53.697423+0000","last_active":"2024-03-24T05:30:19.938289+0000","last_peered":"2024-03-24T05:30:19.938289+0000","last_clean":"2024-03-24T05:30:19.938289+0000","last_became_active":"2024-03-24T05:29:53.696958+0000","last_became_peered":"2024-03-24T05:29:53.696958+0000","last_unstale":"2024-03-24T05:30:19.938289+0000","last_undegraded":"2024-03-24T05:30:19.938289+0000","last_fullsized":"2024-03-24T05:30:19.938289+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:46:11.436240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940235+0000","last_change":"2024-03-24T05:29:55.715462+0000","last_active":"2024-03-24T05:30:19.940235+0000","last_peered":"2024-03-24T05:30:19.940235+0000","last_clean":"2024-03-24T05:30:19.940235+0000","last_became_active":"2024-03-24T05:29:55.714892+0000","last_became_peered":"2024-03-24T05:29:55.714892+0000","last_unstale":"2024-03-24T05:30:19.940235+0000","last_undegraded":"2024-03-24T05:30:19.940235+0000","last_fullsized":"2024-03-24T05:30:19.940235+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:38:04.257745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941906+0000","last_change":"2024-03-24T05:29:57.728305+0000","last_active":"2024-03-24T05:30:19.941906+0000","last_peered":"2024-03-24T05:30:19.941906+0000","last_clean":"2024-03-24T05:30:19.941906+0000","last_became_active":"2024-03-24T05:29:57.727814+0000","last_became_peered":"2024-03-24T05:29:57.727814+0000","last_unstale":"2024-03-24T05:30:19.941906+0000","last_undegraded":"2024-03-24T05:30:19.941906+0000","last_fullsized":"2024-03-24T05:30:19.941906+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:54:26.778782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'30","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937897+0000","last_change":"2024-03-24T05:29:53.697394+0000","last_active":"2024-03-24T05:30:19.937897+0000","last_peered":"2024-03-24T05:30:19.937897+0000","last_clean":"2024-03-24T05:30:19.937897+0000","last_became_active":"2024-03-24T05:29:53.696673+0000","last_became_peered":"2024-03-24T05:29:53.696673+0000","last_unstale":"2024-03-24T05:30:19.937897+0000","last_undegraded":"2024-03-24T05:30:19.937897+0000","last_fullsized":"2024-03-24T05:30:19.937897+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:31:07.573292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938015+0000","last_change":"2024-03-24T05:29:51.691707+0000","last_active":"2024-03-24T05:30:19.938015+0000","last_peered":"2024-03-24T05:30:19.938015+0000","last_clean":"2024-03-24T05:30:19.938015+0000","last_became_active":"2024-03-24T05:29:51.691434+0000","last_became_peered":"2024-03-24T05:29:51.691434+0000","last_unstale":"2024-03-24T05:30:19.938015+0000","last_undegraded":"2024-03-24T05:30:19.938015+0000","last_fullsized":"2024-03-24T05:30:19.938015+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:30.571121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941137+0000","last_change":"2024-03-24T05:29:55.719677+0000","last_active":"2024-03-24T05:30:19.941137+0000","last_peered":"2024-03-24T05:30:19.941137+0000","last_clean":"2024-03-24T05:30:19.941137+0000","last_became_active":"2024-03-24T05:29:55.719074+0000","last_became_peered":"2024-03-24T05:29:55.719074+0000","last_unstale":"2024-03-24T05:30:19.941137+0000","last_undegraded":"2024-03-24T05:30:19.941137+0000","last_fullsized":"2024-03-24T05:30:19.941137+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:09:58.501606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938431+0000","last_change":"2024-03-24T05:29:57.723856+0000","last_active":"2024-03-24T05:30:19.938431+0000","last_peered":"2024-03-24T05:30:19.938431+0000","last_clean":"2024-03-24T05:30:19.938431+0000","last_became_active":"2024-03-24T05:29:57.723213+0000","last_became_peered":"2024-03-24T05:29:57.723213+0000","last_unstale":"2024-03-24T05:30:19.938431+0000","last_undegraded":"2024-03-24T05:30:19.938431+0000","last_fullsized":"2024-03-24T05:30:19.938431+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:52:56.570403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938153+0000","last_change":"2024-03-24T05:29:53.702621+0000","last_active":"2024-03-24T05:30:19.938153+0000","last_peered":"2024-03-24T05:30:19.938153+0000","last_clean":"2024-03-24T05:30:19.938153+0000","last_became_active":"2024-03-24T05:29:53.701835+0000","last_became_peered":"2024-03-24T05:29:53.701835+0000","last_unstale":"2024-03-24T05:30:19.938153+0000","last_undegraded":"2024-03-24T05:30:19.938153+0000","last_fullsized":"2024-03-24T05:30:19.938153+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:04:55.490735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937908+0000","last_change":"2024-03-24T05:29:51.691812+0000","last_active":"2024-03-24T05:30:19.937908+0000","last_peered":"2024-03-24T05:30:19.937908+0000","last_clean":"2024-03-24T05:30:19.937908+0000","last_became_active":"2024-03-24T05:29:51.691360+0000","last_became_peered":"2024-03-24T05:29:51.691360+0000","last_unstale":"2024-03-24T05:30:19.937908+0000","last_undegraded":"2024-03-24T05:30:19.937908+0000","last_fullsized":"2024-03-24T05:30:19.937908+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:28:31.405765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:22.040617+0000","last_change":"2024-03-24T05:29:37.691758+0000","last_active":"2024-03-24T05:30:22.040617+0000","last_peered":"2024-03-24T05:30:22.040617+0000","last_clean":"2024-03-24T05:30:22.040617+0000","last_became_active":"2024-03-24T05:29:37.691498+0000","last_became_peered":"2024-03-24T05:29:37.691498+0000","last_unstale":"2024-03-24T05:30:22.040617+0000","last_undegraded":"2024-03-24T05:30:22.040617+0000","last_fullsized":"2024-03-24T05:30:22.040617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:36.682205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:36.682205+0000","last_clean_scrub_stamp":"2024-03-24T05:29:36.682205+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:16:56.278660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938529+0000","last_change":"2024-03-24T05:29:55.715197+0000","last_active":"2024-03-24T05:30:19.938529+0000","last_peered":"2024-03-24T05:30:19.938529+0000","last_clean":"2024-03-24T05:30:19.938529+0000","last_became_active":"2024-03-24T05:29:55.714317+0000","last_became_peered":"2024-03-24T05:29:55.714317+0000","last_unstale":"2024-03-24T05:30:19.938529+0000","last_undegraded":"2024-03-24T05:30:19.938529+0000","last_fullsized":"2024-03-24T05:30:19.938529+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:50:16.391248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938764+0000","last_change":"2024-03-24T05:29:57.724944+0000","last_active":"2024-03-24T05:30:19.938764+0000","last_peered":"2024-03-24T05:30:19.938764+0000","last_clean":"2024-03-24T05:30:19.938764+0000","last_became_active":"2024-03-24T05:29:57.723662+0000","last_became_peered":"2024-03-24T05:29:57.723662+0000","last_unstale":"2024-03-24T05:30:19.938764+0000","last_undegraded":"2024-03-24T05:30:19.938764+0000","last_fullsized":"2024-03-24T05:30:19.938764+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:00:28.061671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'20","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938219+0000","last_change":"2024-03-24T05:29:53.703755+0000","last_active":"2024-03-24T05:30:19.938219+0000","last_peered":"2024-03-24T05:30:19.938219+0000","last_clean":"2024-03-24T05:30:19.938219+0000","last_became_active":"2024-03-24T05:29:53.703156+0000","last_became_peered":"2024-03-24T05:29:53.703156+0000","last_unstale":"2024-03-24T05:30:19.938219+0000","last_undegraded":"2024-03-24T05:30:19.938219+0000","last_fullsized":"2024-03-24T05:30:19.938219+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:44.462349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.936842+0000","last_change":"2024-03-24T05:29:51.687201+0000","last_active":"2024-03-24T05:30:19.936842+0000","last_peered":"2024-03-24T05:30:19.936842+0000","last_clean":"2024-03-24T05:30:19.936842+0000","last_became_active":"2024-03-24T05:29:51.686074+0000","last_became_peered":"2024-03-24T05:29:51.686074+0000","last_unstale":"2024-03-24T05:30:19.936842+0000","last_undegraded":"2024-03-24T05:30:19.936842+0000","last_fullsized":"2024-03-24T05:30:19.936842+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:56:57.331519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940073+0000","last_change":"2024-03-24T05:29:55.713355+0000","last_active":"2024-03-24T05:30:19.940073+0000","last_peered":"2024-03-24T05:30:19.940073+0000","last_clean":"2024-03-24T05:30:19.940073+0000","last_became_active":"2024-03-24T05:29:55.712999+0000","last_became_peered":"2024-03-24T05:29:55.712999+0000","last_unstale":"2024-03-24T05:30:19.940073+0000","last_undegraded":"2024-03-24T05:30:19.940073+0000","last_fullsized":"2024-03-24T05:30:19.940073+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:44:29.618389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937336+0000","last_change":"2024-03-24T05:29:57.724557+0000","last_active":"2024-03-24T05:30:19.937336+0000","last_peered":"2024-03-24T05:30:19.937336+0000","last_clean":"2024-03-24T05:30:19.937336+0000","last_became_active":"2024-03-24T05:29:57.723634+0000","last_became_peered":"2024-03-24T05:29:57.723634+0000","last_unstale":"2024-03-24T05:30:19.937336+0000","last_undegraded":"2024-03-24T05:30:19.937336+0000","last_fullsized":"2024-03-24T05:30:19.937336+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:10.119301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940942+0000","last_change":"2024-03-24T05:29:51.691613+0000","last_active":"2024-03-24T05:30:19.940942+0000","last_peered":"2024-03-24T05:30:19.940942+0000","last_clean":"2024-03-24T05:30:19.940942+0000","last_became_active":"2024-03-24T05:29:51.690522+0000","last_became_peered":"2024-03-24T05:29:51.690522+0000","last_unstale":"2024-03-24T05:30:19.940942+0000","last_undegraded":"2024-03-24T05:30:19.940942+0000","last_fullsized":"2024-03-24T05:30:19.940942+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:14:17.009626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'36","reported_seq":92,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940429+0000","last_change":"2024-03-24T05:29:53.703629+0000","last_active":"2024-03-24T05:30:19.940429+0000","last_peered":"2024-03-24T05:30:19.940429+0000","last_clean":"2024-03-24T05:30:19.940429+0000","last_became_active":"2024-03-24T05:29:53.703156+0000","last_became_peered":"2024-03-24T05:29:53.703156+0000","last_unstale":"2024-03-24T05:30:19.940429+0000","last_undegraded":"2024-03-24T05:30:19.940429+0000","last_fullsized":"2024-03-24T05:30:19.940429+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:09.753001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938988+0000","last_change":"2024-03-24T05:29:55.713003+0000","last_active":"2024-03-24T05:30:19.938988+0000","last_peered":"2024-03-24T05:30:19.938988+0000","last_clean":"2024-03-24T05:30:19.938988+0000","last_became_active":"2024-03-24T05:29:55.712651+0000","last_became_peered":"2024-03-24T05:29:55.712651+0000","last_unstale":"2024-03-24T05:30:19.938988+0000","last_undegraded":"2024-03-24T05:30:19.938988+0000","last_fullsized":"2024-03-24T05:30:19.938988+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:19:05.588179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.936854+0000","last_change":"2024-03-24T05:29:57.723905+0000","last_active":"2024-03-24T05:30:19.936854+0000","last_peered":"2024-03-24T05:30:19.936854+0000","last_clean":"2024-03-24T05:30:19.936854+0000","last_became_active":"2024-03-24T05:29:57.723296+0000","last_became_peered":"2024-03-24T05:29:57.723296+0000","last_unstale":"2024-03-24T05:30:19.936854+0000","last_undegraded":"2024-03-24T05:30:19.936854+0000","last_fullsized":"2024-03-24T05:30:19.936854+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:36.494956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.942243+0000","last_change":"2024-03-24T05:29:51.692615+0000","last_active":"2024-03-24T05:30:19.942243+0000","last_peered":"2024-03-24T05:30:19.942243+0000","last_clean":"2024-03-24T05:30:19.942243+0000","last_became_active":"2024-03-24T05:29:51.692164+0000","last_became_peered":"2024-03-24T05:29:51.692164+0000","last_unstale":"2024-03-24T05:30:19.942243+0000","last_undegraded":"2024-03-24T05:30:19.942243+0000","last_fullsized":"2024-03-24T05:30:19.942243+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:52.830091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'19","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941687+0000","last_change":"2024-03-24T05:29:53.704033+0000","last_active":"2024-03-24T05:30:19.941687+0000","last_peered":"2024-03-24T05:30:19.941687+0000","last_clean":"2024-03-24T05:30:19.941687+0000","last_became_active":"2024-03-24T05:29:53.703741+0000","last_became_peered":"2024-03-24T05:29:53.703741+0000","last_unstale":"2024-03-24T05:30:19.941687+0000","last_undegraded":"2024-03-24T05:30:19.941687+0000","last_fullsized":"2024-03-24T05:30:19.941687+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:59:19.419593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938381+0000","last_change":"2024-03-24T05:29:55.715604+0000","last_active":"2024-03-24T05:30:19.938381+0000","last_peered":"2024-03-24T05:30:19.938381+0000","last_clean":"2024-03-24T05:30:19.938381+0000","last_became_active":"2024-03-24T05:29:55.714909+0000","last_became_peered":"2024-03-24T05:29:55.714909+0000","last_unstale":"2024-03-24T05:30:19.938381+0000","last_undegraded":"2024-03-24T05:30:19.938381+0000","last_fullsized":"2024-03-24T05:30:19.938381+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:28:22.930231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939057+0000","last_change":"2024-03-24T05:29:57.722194+0000","last_active":"2024-03-24T05:30:19.939057+0000","last_peered":"2024-03-24T05:30:19.939057+0000","last_clean":"2024-03-24T05:30:19.939057+0000","last_became_active":"2024-03-24T05:29:57.721898+0000","last_became_peered":"2024-03-24T05:29:57.721898+0000","last_unstale":"2024-03-24T05:30:19.939057+0000","last_undegraded":"2024-03-24T05:30:19.939057+0000","last_fullsized":"2024-03-24T05:30:19.939057+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:58:27.261499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939310+0000","last_change":"2024-03-24T05:29:51.687572+0000","last_active":"2024-03-24T05:30:19.939310+0000","last_peered":"2024-03-24T05:30:19.939310+0000","last_clean":"2024-03-24T05:30:19.939310+0000","last_became_active":"2024-03-24T05:29:51.686354+0000","last_became_peered":"2024-03-24T05:29:51.686354+0000","last_unstale":"2024-03-24T05:30:19.939310+0000","last_undegraded":"2024-03-24T05:30:19.939310+0000","last_fullsized":"2024-03-24T05:30:19.939310+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:44:29.754475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941882+0000","last_change":"2024-03-24T05:29:53.705417+0000","last_active":"2024-03-24T05:30:19.941882+0000","last_peered":"2024-03-24T05:30:19.941882+0000","last_clean":"2024-03-24T05:30:19.941882+0000","last_became_active":"2024-03-24T05:29:53.705071+0000","last_became_peered":"2024-03-24T05:29:53.705071+0000","last_unstale":"2024-03-24T05:30:19.941882+0000","last_undegraded":"2024-03-24T05:30:19.941882+0000","last_fullsized":"2024-03-24T05:30:19.941882+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:57:15.142142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938173+0000","last_change":"2024-03-24T05:29:55.713051+0000","last_active":"2024-03-24T05:30:19.938173+0000","last_peered":"2024-03-24T05:30:19.938173+0000","last_clean":"2024-03-24T05:30:19.938173+0000","last_became_active":"2024-03-24T05:29:55.712717+0000","last_became_peered":"2024-03-24T05:29:55.712717+0000","last_unstale":"2024-03-24T05:30:19.938173+0000","last_undegraded":"2024-03-24T05:30:19.938173+0000","last_fullsized":"2024-03-24T05:30:19.938173+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:48:15.546019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938860+0000","last_change":"2024-03-24T05:29:57.726760+0000","last_active":"2024-03-24T05:30:19.938860+0000","last_peered":"2024-03-24T05:30:19.938860+0000","last_clean":"2024-03-24T05:30:19.938860+0000","last_became_active":"2024-03-24T05:29:57.725013+0000","last_became_peered":"2024-03-24T05:29:57.725013+0000","last_unstale":"2024-03-24T05:30:19.938860+0000","last_undegraded":"2024-03-24T05:30:19.938860+0000","last_fullsized":"2024-03-24T05:30:19.938860+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:34:04.413542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938481+0000","last_change":"2024-03-24T05:29:55.716980+0000","last_active":"2024-03-24T05:30:19.938481+0000","last_peered":"2024-03-24T05:30:19.938481+0000","last_clean":"2024-03-24T05:30:19.938481+0000","last_became_active":"2024-03-24T05:29:55.715918+0000","last_became_peered":"2024-03-24T05:29:55.715918+0000","last_unstale":"2024-03-24T05:30:19.938481+0000","last_undegraded":"2024-03-24T05:30:19.938481+0000","last_fullsized":"2024-03-24T05:30:19.938481+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:30.346542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.936941+0000","last_change":"2024-03-24T05:29:51.688420+0000","last_active":"2024-03-24T05:30:19.936941+0000","last_peered":"2024-03-24T05:30:19.936941+0000","last_clean":"2024-03-24T05:30:19.936941+0000","last_became_active":"2024-03-24T05:29:51.686381+0000","last_became_peered":"2024-03-24T05:29:51.686381+0000","last_unstale":"2024-03-24T05:30:19.936941+0000","last_undegraded":"2024-03-24T05:30:19.936941+0000","last_fullsized":"2024-03-24T05:30:19.936941+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:35.244318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'24","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939719+0000","last_change":"2024-03-24T05:29:53.698224+0000","last_active":"2024-03-24T05:30:19.939719+0000","last_peered":"2024-03-24T05:30:19.939719+0000","last_clean":"2024-03-24T05:30:19.939719+0000","last_became_active":"2024-03-24T05:29:53.697700+0000","last_became_peered":"2024-03-24T05:29:53.697700+0000","last_unstale":"2024-03-24T05:30:19.939719+0000","last_undegraded":"2024-03-24T05:30:19.939719+0000","last_fullsized":"2024-03-24T05:30:19.939719+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:40:05.984990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941142+0000","last_change":"2024-03-24T05:29:57.724588+0000","last_active":"2024-03-24T05:30:19.941142+0000","last_peered":"2024-03-24T05:30:19.941142+0000","last_clean":"2024-03-24T05:30:19.941142+0000","last_became_active":"2024-03-24T05:29:57.723703+0000","last_became_peered":"2024-03-24T05:29:57.723703+0000","last_unstale":"2024-03-24T05:30:19.941142+0000","last_undegraded":"2024-03-24T05:30:19.941142+0000","last_fullsized":"2024-03-24T05:30:19.941142+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:47:54.467370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940912+0000","last_change":"2024-03-24T05:29:55.719458+0000","last_active":"2024-03-24T05:30:19.940912+0000","last_peered":"2024-03-24T05:30:19.940912+0000","last_clean":"2024-03-24T05:30:19.940912+0000","last_became_active":"2024-03-24T05:29:55.718893+0000","last_became_peered":"2024-03-24T05:29:55.718893+0000","last_unstale":"2024-03-24T05:30:19.940912+0000","last_undegraded":"2024-03-24T05:30:19.940912+0000","last_fullsized":"2024-03-24T05:30:19.940912+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:14:43.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938468+0000","last_change":"2024-03-24T05:29:51.691184+0000","last_active":"2024-03-24T05:30:19.938468+0000","last_peered":"2024-03-24T05:30:19.938468+0000","last_clean":"2024-03-24T05:30:19.938468+0000","last_became_active":"2024-03-24T05:29:51.690900+0000","last_became_peered":"2024-03-24T05:29:51.690900+0000","last_unstale":"2024-03-24T05:30:19.938468+0000","last_undegraded":"2024-03-24T05:30:19.938468+0000","last_fullsized":"2024-03-24T05:30:19.938468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:30:04.668568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'27","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938642+0000","last_change":"2024-03-24T05:29:53.697237+0000","last_active":"2024-03-24T05:30:19.938642+0000","last_peered":"2024-03-24T05:30:19.938642+0000","last_clean":"2024-03-24T05:30:19.938642+0000","last_became_active":"2024-03-24T05:29:53.696612+0000","last_became_peered":"2024-03-24T05:29:53.696612+0000","last_unstale":"2024-03-24T05:30:19.938642+0000","last_undegraded":"2024-03-24T05:30:19.938642+0000","last_fullsized":"2024-03-24T05:30:19.938642+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:03:52.698037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939922+0000","last_change":"2024-03-24T05:29:57.724736+0000","last_active":"2024-03-24T05:30:19.939922+0000","last_peered":"2024-03-24T05:30:19.939922+0000","last_clean":"2024-03-24T05:30:19.939922+0000","last_became_active":"2024-03-24T05:29:57.723857+0000","last_became_peered":"2024-03-24T05:29:57.723857+0000","last_unstale":"2024-03-24T05:30:19.939922+0000","last_undegraded":"2024-03-24T05:30:19.939922+0000","last_fullsized":"2024-03-24T05:30:19.939922+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:52:43.908548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938740+0000","last_change":"2024-03-24T05:29:51.687685+0000","last_active":"2024-03-24T05:30:19.938740+0000","last_peered":"2024-03-24T05:30:19.938740+0000","last_clean":"2024-03-24T05:30:19.938740+0000","last_became_active":"2024-03-24T05:29:51.686005+0000","last_became_peered":"2024-03-24T05:29:51.686005+0000","last_unstale":"2024-03-24T05:30:19.938740+0000","last_undegraded":"2024-03-24T05:30:19.938740+0000","last_fullsized":"2024-03-24T05:30:19.938740+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:45:52.923151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938637+0000","last_change":"2024-03-24T05:29:53.703787+0000","last_active":"2024-03-24T05:30:19.938637+0000","last_peered":"2024-03-24T05:30:19.938637+0000","last_clean":"2024-03-24T05:30:19.938637+0000","last_became_active":"2024-03-24T05:29:53.703332+0000","last_became_peered":"2024-03-24T05:29:53.703332+0000","last_unstale":"2024-03-24T05:30:19.938637+0000","last_undegraded":"2024-03-24T05:30:19.938637+0000","last_fullsized":"2024-03-24T05:30:19.938637+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:31.111846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938507+0000","last_change":"2024-03-24T05:29:55.716849+0000","last_active":"2024-03-24T05:30:19.938507+0000","last_peered":"2024-03-24T05:30:19.938507+0000","last_clean":"2024-03-24T05:30:19.938507+0000","last_became_active":"2024-03-24T05:29:55.715768+0000","last_became_peered":"2024-03-24T05:29:55.715768+0000","last_unstale":"2024-03-24T05:30:19.938507+0000","last_undegraded":"2024-03-24T05:30:19.938507+0000","last_fullsized":"2024-03-24T05:30:19.938507+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:48:05.840788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939858+0000","last_change":"2024-03-24T05:29:57.728423+0000","last_active":"2024-03-24T05:30:19.939858+0000","last_peered":"2024-03-24T05:30:19.939858+0000","last_clean":"2024-03-24T05:30:19.939858+0000","last_became_active":"2024-03-24T05:29:57.728106+0000","last_became_peered":"2024-03-24T05:29:57.728106+0000","last_unstale":"2024-03-24T05:30:19.939858+0000","last_undegraded":"2024-03-24T05:30:19.939858+0000","last_fullsized":"2024-03-24T05:30:19.939858+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:12:36.838301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938468+0000","last_change":"2024-03-24T05:29:55.715615+0000","last_active":"2024-03-24T05:30:19.938468+0000","last_peered":"2024-03-24T05:30:19.938468+0000","last_clean":"2024-03-24T05:30:19.938468+0000","last_became_active":"2024-03-24T05:29:55.715210+0000","last_became_peered":"2024-03-24T05:29:55.715210+0000","last_unstale":"2024-03-24T05:30:19.938468+0000","last_undegraded":"2024-03-24T05:30:19.938468+0000","last_fullsized":"2024-03-24T05:30:19.938468+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:29:07.973624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939195+0000","last_change":"2024-03-24T05:29:51.688997+0000","last_active":"2024-03-24T05:30:19.939195+0000","last_peered":"2024-03-24T05:30:19.939195+0000","last_clean":"2024-03-24T05:30:19.939195+0000","last_became_active":"2024-03-24T05:29:51.688525+0000","last_became_peered":"2024-03-24T05:29:51.688525+0000","last_unstale":"2024-03-24T05:30:19.939195+0000","last_undegraded":"2024-03-24T05:30:19.939195+0000","last_fullsized":"2024-03-24T05:30:19.939195+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:26:08.514377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'25","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937550+0000","last_change":"2024-03-24T05:29:53.697062+0000","last_active":"2024-03-24T05:30:19.937550+0000","last_peered":"2024-03-24T05:30:19.937550+0000","last_clean":"2024-03-24T05:30:19.937550+0000","last_became_active":"2024-03-24T05:29:53.696765+0000","last_became_peered":"2024-03-24T05:29:53.696765+0000","last_unstale":"2024-03-24T05:30:19.937550+0000","last_undegraded":"2024-03-24T05:30:19.937550+0000","last_fullsized":"2024-03-24T05:30:19.937550+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:11:36.773653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938422+0000","last_change":"2024-03-24T05:29:57.724736+0000","last_active":"2024-03-24T05:30:19.938422+0000","last_peered":"2024-03-24T05:30:19.938422+0000","last_clean":"2024-03-24T05:30:19.938422+0000","last_became_active":"2024-03-24T05:29:57.723840+0000","last_became_peered":"2024-03-24T05:29:57.723840+0000","last_unstale":"2024-03-24T05:30:19.938422+0000","last_undegraded":"2024-03-24T05:30:19.938422+0000","last_fullsized":"2024-03-24T05:30:19.938422+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:02:24.814339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939597+0000","last_change":"2024-03-24T05:29:51.687476+0000","last_active":"2024-03-24T05:30:19.939597+0000","last_peered":"2024-03-24T05:30:19.939597+0000","last_clean":"2024-03-24T05:30:19.939597+0000","last_became_active":"2024-03-24T05:29:51.686500+0000","last_became_peered":"2024-03-24T05:29:51.686500+0000","last_unstale":"2024-03-24T05:30:19.939597+0000","last_undegraded":"2024-03-24T05:30:19.939597+0000","last_fullsized":"2024-03-24T05:30:19.939597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:50:25.773727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938961+0000","last_change":"2024-03-24T05:29:53.697001+0000","last_active":"2024-03-24T05:30:19.938961+0000","last_peered":"2024-03-24T05:30:19.938961+0000","last_clean":"2024-03-24T05:30:19.938961+0000","last_became_active":"2024-03-24T05:29:53.696653+0000","last_became_peered":"2024-03-24T05:29:53.696653+0000","last_unstale":"2024-03-24T05:30:19.938961+0000","last_undegraded":"2024-03-24T05:30:19.938961+0000","last_fullsized":"2024-03-24T05:30:19.938961+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:09:48.897812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937238+0000","last_change":"2024-03-24T05:29:55.715421+0000","last_active":"2024-03-24T05:30:19.937238+0000","last_peered":"2024-03-24T05:30:19.937238+0000","last_clean":"2024-03-24T05:30:19.937238+0000","last_became_active":"2024-03-24T05:29:55.714893+0000","last_became_peered":"2024-03-24T05:29:55.714893+0000","last_unstale":"2024-03-24T05:30:19.937238+0000","last_undegraded":"2024-03-24T05:30:19.937238+0000","last_fullsized":"2024-03-24T05:30:19.937238+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:27:00.157605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937742+0000","last_change":"2024-03-24T05:29:57.722165+0000","last_active":"2024-03-24T05:30:19.937742+0000","last_peered":"2024-03-24T05:30:19.937742+0000","last_clean":"2024-03-24T05:30:19.937742+0000","last_became_active":"2024-03-24T05:29:57.721846+0000","last_became_peered":"2024-03-24T05:29:57.721846+0000","last_unstale":"2024-03-24T05:30:19.937742+0000","last_undegraded":"2024-03-24T05:30:19.937742+0000","last_fullsized":"2024-03-24T05:30:19.937742+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:37:57.188558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940071+0000","last_change":"2024-03-24T05:29:51.691612+0000","last_active":"2024-03-24T05:30:19.940071+0000","last_peered":"2024-03-24T05:30:19.940071+0000","last_clean":"2024-03-24T05:30:19.940071+0000","last_became_active":"2024-03-24T05:29:51.690596+0000","last_became_peered":"2024-03-24T05:29:51.690596+0000","last_unstale":"2024-03-24T05:30:19.940071+0000","last_undegraded":"2024-03-24T05:30:19.940071+0000","last_fullsized":"2024-03-24T05:30:19.940071+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:56.485216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938971+0000","last_change":"2024-03-24T05:29:53.699585+0000","last_active":"2024-03-24T05:30:19.938971+0000","last_peered":"2024-03-24T05:30:19.938971+0000","last_clean":"2024-03-24T05:30:19.938971+0000","last_became_active":"2024-03-24T05:29:53.699232+0000","last_became_peered":"2024-03-24T05:29:53.699232+0000","last_unstale":"2024-03-24T05:30:19.938971+0000","last_undegraded":"2024-03-24T05:30:19.938971+0000","last_fullsized":"2024-03-24T05:30:19.938971+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:36:16.813779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938434+0000","last_change":"2024-03-24T05:29:55.715215+0000","last_active":"2024-03-24T05:30:19.938434+0000","last_peered":"2024-03-24T05:30:19.938434+0000","last_clean":"2024-03-24T05:30:19.938434+0000","last_became_active":"2024-03-24T05:29:55.714875+0000","last_became_peered":"2024-03-24T05:29:55.714875+0000","last_unstale":"2024-03-24T05:30:19.938434+0000","last_undegraded":"2024-03-24T05:30:19.938434+0000","last_fullsized":"2024-03-24T05:30:19.938434+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:43.320858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.936695+0000","last_change":"2024-03-24T05:29:57.724720+0000","last_active":"2024-03-24T05:30:19.936695+0000","last_peered":"2024-03-24T05:30:19.936695+0000","last_clean":"2024-03-24T05:30:19.936695+0000","last_became_active":"2024-03-24T05:29:57.723868+0000","last_became_peered":"2024-03-24T05:29:57.723868+0000","last_unstale":"2024-03-24T05:30:19.936695+0000","last_undegraded":"2024-03-24T05:30:19.936695+0000","last_fullsized":"2024-03-24T05:30:19.936695+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:01:21.433075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.942082+0000","last_change":"2024-03-24T05:29:51.692856+0000","last_active":"2024-03-24T05:30:19.942082+0000","last_peered":"2024-03-24T05:30:19.942082+0000","last_clean":"2024-03-24T05:30:19.942082+0000","last_became_active":"2024-03-24T05:29:51.692305+0000","last_became_peered":"2024-03-24T05:29:51.692305+0000","last_unstale":"2024-03-24T05:30:19.942082+0000","last_undegraded":"2024-03-24T05:30:19.942082+0000","last_fullsized":"2024-03-24T05:30:19.942082+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:27.885189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939478+0000","last_change":"2024-03-24T05:29:53.698436+0000","last_active":"2024-03-24T05:30:19.939478+0000","last_peered":"2024-03-24T05:30:19.939478+0000","last_clean":"2024-03-24T05:30:19.939478+0000","last_became_active":"2024-03-24T05:29:53.698128+0000","last_became_peered":"2024-03-24T05:29:53.698128+0000","last_unstale":"2024-03-24T05:30:19.939478+0000","last_undegraded":"2024-03-24T05:30:19.939478+0000","last_fullsized":"2024-03-24T05:30:19.939478+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:26:45.435035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938869+0000","last_change":"2024-03-24T05:29:55.716929+0000","last_active":"2024-03-24T05:30:19.938869+0000","last_peered":"2024-03-24T05:30:19.938869+0000","last_clean":"2024-03-24T05:30:19.938869+0000","last_became_active":"2024-03-24T05:29:55.716550+0000","last_became_peered":"2024-03-24T05:29:55.716550+0000","last_unstale":"2024-03-24T05:30:19.938869+0000","last_undegraded":"2024-03-24T05:30:19.938869+0000","last_fullsized":"2024-03-24T05:30:19.938869+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:02.900015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":587,"ondisk_log_size":587,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411340,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738391,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T05:30:31.272 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T05:30:31.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:30 smithi063 ceph-mon[30595]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-24T05:30:31.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:30 smithi063 ceph-mon[30595]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T05:30:31.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:30 smithi143 ceph-mon[30288]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-24T05:30:31.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:30 smithi143 ceph-mon[30288]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T05:30:31.514 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:30:33.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:33 smithi063 ceph-mon[30595]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T05:30:33.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:33 smithi143 ceph-mon[30288]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T05:30:33.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:33 smithi023 ceph-mon[25869]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T05:30:34.963 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:30:34.966 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-24T05:30:35.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:35 smithi063 ceph-mon[30595]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T05:30:35.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:35 smithi143 ceph-mon[30288]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T05:30:35.311 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:35 smithi023 ceph-mon[25869]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T05:30:35.663 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-03-24T05:30:33.950494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":1262,"num_read_kb":1077,"num_write":889,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":867,"ondisk_log_size":867,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96612,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126556,"statfs":{"total":287972524032,"available":287873593344,"internally_reserved":0,"allocated":3158016,"data_stored":2100351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002664"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937329+0000","last_change":"2024-03-24T05:29:57.725554+0000","last_active":"2024-03-24T05:30:19.937329+0000","last_peered":"2024-03-24T05:30:19.937329+0000","last_clean":"2024-03-24T05:30:19.937329+0000","last_became_active":"2024-03-24T05:29:57.724358+0000","last_became_peered":"2024-03-24T05:29:57.724358+0000","last_unstale":"2024-03-24T05:30:19.937329+0000","last_undegraded":"2024-03-24T05:30:19.937329+0000","last_fullsized":"2024-03-24T05:30:19.937329+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:50:53.604623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937930+0000","last_change":"2024-03-24T05:29:55.712962+0000","last_active":"2024-03-24T05:30:19.937930+0000","last_peered":"2024-03-24T05:30:19.937930+0000","last_clean":"2024-03-24T05:30:19.937930+0000","last_became_active":"2024-03-24T05:29:55.712686+0000","last_became_peered":"2024-03-24T05:29:55.712686+0000","last_unstale":"2024-03-24T05:30:19.937930+0000","last_undegraded":"2024-03-24T05:30:19.937930+0000","last_fullsized":"2024-03-24T05:30:19.937930+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:56:30.794523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941461+0000","last_change":"2024-03-24T05:29:51.691913+0000","last_active":"2024-03-24T05:30:19.941461+0000","last_peered":"2024-03-24T05:30:19.941461+0000","last_clean":"2024-03-24T05:30:19.941461+0000","last_became_active":"2024-03-24T05:29:51.691620+0000","last_became_peered":"2024-03-24T05:29:51.691620+0000","last_unstale":"2024-03-24T05:30:19.941461+0000","last_undegraded":"2024-03-24T05:30:19.941461+0000","last_fullsized":"2024-03-24T05:30:19.941461+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:15.001586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.942027+0000","last_change":"2024-03-24T05:29:53.703805+0000","last_active":"2024-03-24T05:30:19.942027+0000","last_peered":"2024-03-24T05:30:19.942027+0000","last_clean":"2024-03-24T05:30:19.942027+0000","last_became_active":"2024-03-24T05:29:53.703408+0000","last_became_peered":"2024-03-24T05:29:53.703408+0000","last_unstale":"2024-03-24T05:30:19.942027+0000","last_undegraded":"2024-03-24T05:30:19.942027+0000","last_fullsized":"2024-03-24T05:30:19.942027+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:40:42.134534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938367+0000","last_change":"2024-03-24T05:29:57.728119+0000","last_active":"2024-03-24T05:30:19.938367+0000","last_peered":"2024-03-24T05:30:19.938367+0000","last_clean":"2024-03-24T05:30:19.938367+0000","last_became_active":"2024-03-24T05:29:57.727599+0000","last_became_peered":"2024-03-24T05:29:57.727599+0000","last_unstale":"2024-03-24T05:30:19.938367+0000","last_undegraded":"2024-03-24T05:30:19.938367+0000","last_fullsized":"2024-03-24T05:30:19.938367+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:12:19.388327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937035+0000","last_change":"2024-03-24T05:29:55.717109+0000","last_active":"2024-03-24T05:30:19.937035+0000","last_peered":"2024-03-24T05:30:19.937035+0000","last_clean":"2024-03-24T05:30:19.937035+0000","last_became_active":"2024-03-24T05:29:55.715969+0000","last_became_peered":"2024-03-24T05:29:55.715969+0000","last_unstale":"2024-03-24T05:30:19.937035+0000","last_undegraded":"2024-03-24T05:30:19.937035+0000","last_fullsized":"2024-03-24T05:30:19.937035+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:58:35.185051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.942184+0000","last_change":"2024-03-24T05:29:51.692863+0000","last_active":"2024-03-24T05:30:19.942184+0000","last_peered":"2024-03-24T05:30:19.942184+0000","last_clean":"2024-03-24T05:30:19.942184+0000","last_became_active":"2024-03-24T05:29:51.692559+0000","last_became_peered":"2024-03-24T05:29:51.692559+0000","last_unstale":"2024-03-24T05:30:19.942184+0000","last_undegraded":"2024-03-24T05:30:19.942184+0000","last_fullsized":"2024-03-24T05:30:19.942184+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:58.106532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941005+0000","last_change":"2024-03-24T05:29:53.705316+0000","last_active":"2024-03-24T05:30:19.941005+0000","last_peered":"2024-03-24T05:30:19.941005+0000","last_clean":"2024-03-24T05:30:19.941005+0000","last_became_active":"2024-03-24T05:29:53.704880+0000","last_became_peered":"2024-03-24T05:29:53.704880+0000","last_unstale":"2024-03-24T05:30:19.941005+0000","last_undegraded":"2024-03-24T05:30:19.941005+0000","last_fullsized":"2024-03-24T05:30:19.941005+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:58:35.687715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938039+0000","last_change":"2024-03-24T05:29:57.723671+0000","last_active":"2024-03-24T05:30:19.938039+0000","last_peered":"2024-03-24T05:30:19.938039+0000","last_clean":"2024-03-24T05:30:19.938039+0000","last_became_active":"2024-03-24T05:29:57.723133+0000","last_became_peered":"2024-03-24T05:29:57.723133+0000","last_unstale":"2024-03-24T05:30:19.938039+0000","last_undegraded":"2024-03-24T05:30:19.938039+0000","last_fullsized":"2024-03-24T05:30:19.938039+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:38:56.045037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937550+0000","last_change":"2024-03-24T05:29:55.717322+0000","last_active":"2024-03-24T05:30:19.937550+0000","last_peered":"2024-03-24T05:30:19.937550+0000","last_clean":"2024-03-24T05:30:19.937550+0000","last_became_active":"2024-03-24T05:29:55.716932+0000","last_became_peered":"2024-03-24T05:29:55.716932+0000","last_unstale":"2024-03-24T05:30:19.937550+0000","last_undegraded":"2024-03-24T05:30:19.937550+0000","last_fullsized":"2024-03-24T05:30:19.937550+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:38:53.342699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941119+0000","last_change":"2024-03-24T05:29:51.690829+0000","last_active":"2024-03-24T05:30:19.941119+0000","last_peered":"2024-03-24T05:30:19.941119+0000","last_clean":"2024-03-24T05:30:19.941119+0000","last_became_active":"2024-03-24T05:29:51.688559+0000","last_became_peered":"2024-03-24T05:29:51.688559+0000","last_unstale":"2024-03-24T05:30:19.941119+0000","last_undegraded":"2024-03-24T05:30:19.941119+0000","last_fullsized":"2024-03-24T05:30:19.941119+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:03:33.365876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939106+0000","last_change":"2024-03-24T05:29:53.697105+0000","last_active":"2024-03-24T05:30:19.939106+0000","last_peered":"2024-03-24T05:30:19.939106+0000","last_clean":"2024-03-24T05:30:19.939106+0000","last_became_active":"2024-03-24T05:29:53.696742+0000","last_became_peered":"2024-03-24T05:29:53.696742+0000","last_unstale":"2024-03-24T05:30:19.939106+0000","last_undegraded":"2024-03-24T05:30:19.939106+0000","last_fullsized":"2024-03-24T05:30:19.939106+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:21:08.421214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937443+0000","last_change":"2024-03-24T05:29:57.725621+0000","last_active":"2024-03-24T05:30:19.937443+0000","last_peered":"2024-03-24T05:30:19.937443+0000","last_clean":"2024-03-24T05:30:19.937443+0000","last_became_active":"2024-03-24T05:29:57.723737+0000","last_became_peered":"2024-03-24T05:29:57.723737+0000","last_unstale":"2024-03-24T05:30:19.937443+0000","last_undegraded":"2024-03-24T05:30:19.937443+0000","last_fullsized":"2024-03-24T05:30:19.937443+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:37:20.857940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937131+0000","last_change":"2024-03-24T05:29:55.715512+0000","last_active":"2024-03-24T05:30:19.937131+0000","last_peered":"2024-03-24T05:30:19.937131+0000","last_clean":"2024-03-24T05:30:19.937131+0000","last_became_active":"2024-03-24T05:29:55.714880+0000","last_became_peered":"2024-03-24T05:29:55.714880+0000","last_unstale":"2024-03-24T05:30:19.937131+0000","last_undegraded":"2024-03-24T05:30:19.937131+0000","last_fullsized":"2024-03-24T05:30:19.937131+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:02.629129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939316+0000","last_change":"2024-03-24T05:29:51.688533+0000","last_active":"2024-03-24T05:30:19.939316+0000","last_peered":"2024-03-24T05:30:19.939316+0000","last_clean":"2024-03-24T05:30:19.939316+0000","last_became_active":"2024-03-24T05:29:51.687764+0000","last_became_peered":"2024-03-24T05:29:51.687764+0000","last_unstale":"2024-03-24T05:30:19.939316+0000","last_undegraded":"2024-03-24T05:30:19.939316+0000","last_fullsized":"2024-03-24T05:30:19.939316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:02:51.403460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'18","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941066+0000","last_change":"2024-03-24T05:29:53.698767+0000","last_active":"2024-03-24T05:30:19.941066+0000","last_peered":"2024-03-24T05:30:19.941066+0000","last_clean":"2024-03-24T05:30:19.941066+0000","last_became_active":"2024-03-24T05:29:53.698257+0000","last_became_peered":"2024-03-24T05:29:53.698257+0000","last_unstale":"2024-03-24T05:30:19.941066+0000","last_undegraded":"2024-03-24T05:30:19.941066+0000","last_fullsized":"2024-03-24T05:30:19.941066+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:45:55.798554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938396+0000","last_change":"2024-03-24T05:29:57.725054+0000","last_active":"2024-03-24T05:30:19.938396+0000","last_peered":"2024-03-24T05:30:19.938396+0000","last_clean":"2024-03-24T05:30:19.938396+0000","last_became_active":"2024-03-24T05:29:57.723935+0000","last_became_peered":"2024-03-24T05:29:57.723935+0000","last_unstale":"2024-03-24T05:30:19.938396+0000","last_undegraded":"2024-03-24T05:30:19.938396+0000","last_fullsized":"2024-03-24T05:30:19.938396+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:45:20.040217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939946+0000","last_change":"2024-03-24T05:29:55.716731+0000","last_active":"2024-03-24T05:30:19.939946+0000","last_peered":"2024-03-24T05:30:19.939946+0000","last_clean":"2024-03-24T05:30:19.939946+0000","last_became_active":"2024-03-24T05:29:55.716303+0000","last_became_peered":"2024-03-24T05:29:55.716303+0000","last_unstale":"2024-03-24T05:30:19.939946+0000","last_undegraded":"2024-03-24T05:30:19.939946+0000","last_fullsized":"2024-03-24T05:30:19.939946+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:46:09.397749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939409+0000","last_change":"2024-03-24T05:29:51.688388+0000","last_active":"2024-03-24T05:30:19.939409+0000","last_peered":"2024-03-24T05:30:19.939409+0000","last_clean":"2024-03-24T05:30:19.939409+0000","last_became_active":"2024-03-24T05:29:51.686710+0000","last_became_peered":"2024-03-24T05:29:51.686710+0000","last_unstale":"2024-03-24T05:30:19.939409+0000","last_undegraded":"2024-03-24T05:30:19.939409+0000","last_fullsized":"2024-03-24T05:30:19.939409+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:14.513640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'21","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938806+0000","last_change":"2024-03-24T05:29:53.700518+0000","last_active":"2024-03-24T05:30:19.938806+0000","last_peered":"2024-03-24T05:30:19.938806+0000","last_clean":"2024-03-24T05:30:19.938806+0000","last_became_active":"2024-03-24T05:29:53.700218+0000","last_became_peered":"2024-03-24T05:29:53.700218+0000","last_unstale":"2024-03-24T05:30:19.938806+0000","last_undegraded":"2024-03-24T05:30:19.938806+0000","last_fullsized":"2024-03-24T05:30:19.938806+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:12:06.439923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939906+0000","last_change":"2024-03-24T05:29:57.729024+0000","last_active":"2024-03-24T05:30:19.939906+0000","last_peered":"2024-03-24T05:30:19.939906+0000","last_clean":"2024-03-24T05:30:19.939906+0000","last_became_active":"2024-03-24T05:29:57.728713+0000","last_became_peered":"2024-03-24T05:29:57.728713+0000","last_unstale":"2024-03-24T05:30:19.939906+0000","last_undegraded":"2024-03-24T05:30:19.939906+0000","last_fullsized":"2024-03-24T05:30:19.939906+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:21.817744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939082+0000","last_change":"2024-03-24T05:29:55.713007+0000","last_active":"2024-03-24T05:30:19.939082+0000","last_peered":"2024-03-24T05:30:19.939082+0000","last_clean":"2024-03-24T05:30:19.939082+0000","last_became_active":"2024-03-24T05:29:55.712701+0000","last_became_peered":"2024-03-24T05:29:55.712701+0000","last_unstale":"2024-03-24T05:30:19.939082+0000","last_undegraded":"2024-03-24T05:30:19.939082+0000","last_fullsized":"2024-03-24T05:30:19.939082+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:44:23.593104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937360+0000","last_change":"2024-03-24T05:29:51.687604+0000","last_active":"2024-03-24T05:30:19.937360+0000","last_peered":"2024-03-24T05:30:19.937360+0000","last_clean":"2024-03-24T05:30:19.937360+0000","last_became_active":"2024-03-24T05:29:51.687205+0000","last_became_peered":"2024-03-24T05:29:51.687205+0000","last_unstale":"2024-03-24T05:30:19.937360+0000","last_undegraded":"2024-03-24T05:30:19.937360+0000","last_fullsized":"2024-03-24T05:30:19.937360+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:05:04.180724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'11","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939055+0000","last_change":"2024-03-24T05:29:53.700618+0000","last_active":"2024-03-24T05:30:19.939055+0000","last_peered":"2024-03-24T05:30:19.939055+0000","last_clean":"2024-03-24T05:30:19.939055+0000","last_became_active":"2024-03-24T05:29:53.700243+0000","last_became_peered":"2024-03-24T05:29:53.700243+0000","last_unstale":"2024-03-24T05:30:19.939055+0000","last_undegraded":"2024-03-24T05:30:19.939055+0000","last_fullsized":"2024-03-24T05:30:19.939055+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:03:29.518774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939201+0000","last_change":"2024-03-24T05:29:57.728603+0000","last_active":"2024-03-24T05:30:19.939201+0000","last_peered":"2024-03-24T05:30:19.939201+0000","last_clean":"2024-03-24T05:30:19.939201+0000","last_became_active":"2024-03-24T05:29:57.728097+0000","last_became_peered":"2024-03-24T05:29:57.728097+0000","last_unstale":"2024-03-24T05:30:19.939201+0000","last_undegraded":"2024-03-24T05:30:19.939201+0000","last_fullsized":"2024-03-24T05:30:19.939201+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:52.482520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:21.736197+0000","last_change":"2024-03-24T05:29:55.717112+0000","last_active":"2024-03-24T05:30:21.736197+0000","last_peered":"2024-03-24T05:30:21.736197+0000","last_clean":"2024-03-24T05:30:21.736197+0000","last_became_active":"2024-03-24T05:29:55.715949+0000","last_became_peered":"2024-03-24T05:29:55.715949+0000","last_unstale":"2024-03-24T05:30:21.736197+0000","last_undegraded":"2024-03-24T05:30:21.736197+0000","last_fullsized":"2024-03-24T05:30:21.736197+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:53:05.607989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938724+0000","last_change":"2024-03-24T05:29:51.691411+0000","last_active":"2024-03-24T05:30:19.938724+0000","last_peered":"2024-03-24T05:30:19.938724+0000","last_clean":"2024-03-24T05:30:19.938724+0000","last_became_active":"2024-03-24T05:29:51.691023+0000","last_became_peered":"2024-03-24T05:29:51.691023+0000","last_unstale":"2024-03-24T05:30:19.938724+0000","last_undegraded":"2024-03-24T05:30:19.938724+0000","last_fullsized":"2024-03-24T05:30:19.938724+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:04:48.345391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937742+0000","last_change":"2024-03-24T05:29:53.702446+0000","last_active":"2024-03-24T05:30:19.937742+0000","last_peered":"2024-03-24T05:30:19.937742+0000","last_clean":"2024-03-24T05:30:19.937742+0000","last_became_active":"2024-03-24T05:29:53.702144+0000","last_became_peered":"2024-03-24T05:29:53.702144+0000","last_unstale":"2024-03-24T05:30:19.937742+0000","last_undegraded":"2024-03-24T05:30:19.937742+0000","last_fullsized":"2024-03-24T05:30:19.937742+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:59:30.591850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938656+0000","last_change":"2024-03-24T05:29:57.726574+0000","last_active":"2024-03-24T05:30:19.938656+0000","last_peered":"2024-03-24T05:30:19.938656+0000","last_clean":"2024-03-24T05:30:19.938656+0000","last_became_active":"2024-03-24T05:29:57.724489+0000","last_became_peered":"2024-03-24T05:29:57.724489+0000","last_unstale":"2024-03-24T05:30:19.938656+0000","last_undegraded":"2024-03-24T05:30:19.938656+0000","last_fullsized":"2024-03-24T05:30:19.938656+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:18.385378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941700+0000","last_change":"2024-03-24T05:29:55.715847+0000","last_active":"2024-03-24T05:30:19.941700+0000","last_peered":"2024-03-24T05:30:19.941700+0000","last_clean":"2024-03-24T05:30:19.941700+0000","last_became_active":"2024-03-24T05:29:55.715497+0000","last_became_peered":"2024-03-24T05:29:55.715497+0000","last_unstale":"2024-03-24T05:30:19.941700+0000","last_undegraded":"2024-03-24T05:30:19.941700+0000","last_fullsized":"2024-03-24T05:30:19.941700+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:12:37.665790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938380+0000","last_change":"2024-03-24T05:29:51.691384+0000","last_active":"2024-03-24T05:30:19.938380+0000","last_peered":"2024-03-24T05:30:19.938380+0000","last_clean":"2024-03-24T05:30:19.938380+0000","last_became_active":"2024-03-24T05:29:51.689458+0000","last_became_peered":"2024-03-24T05:29:51.689458+0000","last_unstale":"2024-03-24T05:30:19.938380+0000","last_undegraded":"2024-03-24T05:30:19.938380+0000","last_fullsized":"2024-03-24T05:30:19.938380+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:52:03.687301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'39","reported_seq":96,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937967+0000","last_change":"2024-03-24T05:29:53.702302+0000","last_active":"2024-03-24T05:30:19.937967+0000","last_peered":"2024-03-24T05:30:19.937967+0000","last_clean":"2024-03-24T05:30:19.937967+0000","last_became_active":"2024-03-24T05:29:53.701917+0000","last_became_peered":"2024-03-24T05:29:53.701917+0000","last_unstale":"2024-03-24T05:30:19.937967+0000","last_undegraded":"2024-03-24T05:30:19.937967+0000","last_fullsized":"2024-03-24T05:30:19.937967+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:31.597068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937217+0000","last_change":"2024-03-24T05:29:57.723600+0000","last_active":"2024-03-24T05:30:19.937217+0000","last_peered":"2024-03-24T05:30:19.937217+0000","last_clean":"2024-03-24T05:30:19.937217+0000","last_became_active":"2024-03-24T05:29:57.722672+0000","last_became_peered":"2024-03-24T05:29:57.722672+0000","last_unstale":"2024-03-24T05:30:19.937217+0000","last_undegraded":"2024-03-24T05:30:19.937217+0000","last_fullsized":"2024-03-24T05:30:19.937217+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:05:37.886835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937827+0000","last_change":"2024-03-24T05:29:55.712734+0000","last_active":"2024-03-24T05:30:19.937827+0000","last_peered":"2024-03-24T05:30:19.937827+0000","last_clean":"2024-03-24T05:30:19.937827+0000","last_became_active":"2024-03-24T05:29:55.712361+0000","last_became_peered":"2024-03-24T05:29:55.712361+0000","last_unstale":"2024-03-24T05:30:19.937827+0000","last_undegraded":"2024-03-24T05:30:19.937827+0000","last_fullsized":"2024-03-24T05:30:19.937827+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:55:28.982750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939645+0000","last_change":"2024-03-24T05:29:51.687454+0000","last_active":"2024-03-24T05:30:19.939645+0000","last_peered":"2024-03-24T05:30:19.939645+0000","last_clean":"2024-03-24T05:30:19.939645+0000","last_became_active":"2024-03-24T05:29:51.686318+0000","last_became_peered":"2024-03-24T05:29:51.686318+0000","last_unstale":"2024-03-24T05:30:19.939645+0000","last_undegraded":"2024-03-24T05:30:19.939645+0000","last_fullsized":"2024-03-24T05:30:19.939645+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:55:43.706601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'21","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939175+0000","last_change":"2024-03-24T05:29:53.698120+0000","last_active":"2024-03-24T05:30:19.939175+0000","last_peered":"2024-03-24T05:30:19.939175+0000","last_clean":"2024-03-24T05:30:19.939175+0000","last_became_active":"2024-03-24T05:29:53.697313+0000","last_became_peered":"2024-03-24T05:29:53.697313+0000","last_unstale":"2024-03-24T05:30:19.939175+0000","last_undegraded":"2024-03-24T05:30:19.939175+0000","last_fullsized":"2024-03-24T05:30:19.939175+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:37.043805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937693+0000","last_change":"2024-03-24T05:29:57.724474+0000","last_active":"2024-03-24T05:30:19.937693+0000","last_peered":"2024-03-24T05:30:19.937693+0000","last_clean":"2024-03-24T05:30:19.937693+0000","last_became_active":"2024-03-24T05:29:57.723522+0000","last_became_peered":"2024-03-24T05:29:57.723522+0000","last_unstale":"2024-03-24T05:30:19.937693+0000","last_undegraded":"2024-03-24T05:30:19.937693+0000","last_fullsized":"2024-03-24T05:30:19.937693+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:58.222918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938232+0000","last_change":"2024-03-24T05:29:55.716186+0000","last_active":"2024-03-24T05:30:19.938232+0000","last_peered":"2024-03-24T05:30:19.938232+0000","last_clean":"2024-03-24T05:30:19.938232+0000","last_became_active":"2024-03-24T05:29:55.715910+0000","last_became_peered":"2024-03-24T05:29:55.715910+0000","last_unstale":"2024-03-24T05:30:19.938232+0000","last_undegraded":"2024-03-24T05:30:19.938232+0000","last_fullsized":"2024-03-24T05:30:19.938232+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:02:06.223913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938741+0000","last_change":"2024-03-24T05:29:51.685304+0000","last_active":"2024-03-24T05:30:19.938741+0000","last_peered":"2024-03-24T05:30:19.938741+0000","last_clean":"2024-03-24T05:30:19.938741+0000","last_became_active":"2024-03-24T05:29:51.684613+0000","last_became_peered":"2024-03-24T05:29:51.684613+0000","last_unstale":"2024-03-24T05:30:19.938741+0000","last_undegraded":"2024-03-24T05:30:19.938741+0000","last_fullsized":"2024-03-24T05:30:19.938741+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:24.326969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'12","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941225+0000","last_change":"2024-03-24T05:29:53.703499+0000","last_active":"2024-03-24T05:30:19.941225+0000","last_peered":"2024-03-24T05:30:19.941225+0000","last_clean":"2024-03-24T05:30:19.941225+0000","last_became_active":"2024-03-24T05:29:53.702938+0000","last_became_peered":"2024-03-24T05:29:53.702938+0000","last_unstale":"2024-03-24T05:30:19.941225+0000","last_undegraded":"2024-03-24T05:30:19.941225+0000","last_fullsized":"2024-03-24T05:30:19.941225+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:05.424232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938202+0000","last_change":"2024-03-24T05:29:57.723093+0000","last_active":"2024-03-24T05:30:19.938202+0000","last_peered":"2024-03-24T05:30:19.938202+0000","last_clean":"2024-03-24T05:30:19.938202+0000","last_became_active":"2024-03-24T05:29:57.722791+0000","last_became_peered":"2024-03-24T05:29:57.722791+0000","last_unstale":"2024-03-24T05:30:19.938202+0000","last_undegraded":"2024-03-24T05:30:19.938202+0000","last_fullsized":"2024-03-24T05:30:19.938202+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:55:37.633776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937808+0000","last_change":"2024-03-24T05:29:55.717043+0000","last_active":"2024-03-24T05:30:19.937808+0000","last_peered":"2024-03-24T05:30:19.937808+0000","last_clean":"2024-03-24T05:30:19.937808+0000","last_became_active":"2024-03-24T05:29:55.715968+0000","last_became_peered":"2024-03-24T05:29:55.715968+0000","last_unstale":"2024-03-24T05:30:19.937808+0000","last_undegraded":"2024-03-24T05:30:19.937808+0000","last_fullsized":"2024-03-24T05:30:19.937808+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:29:40.714692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941013+0000","last_change":"2024-03-24T05:29:51.690997+0000","last_active":"2024-03-24T05:30:19.941013+0000","last_peered":"2024-03-24T05:30:19.941013+0000","last_clean":"2024-03-24T05:30:19.941013+0000","last_became_active":"2024-03-24T05:29:51.689504+0000","last_became_peered":"2024-03-24T05:29:51.689504+0000","last_unstale":"2024-03-24T05:30:19.941013+0000","last_undegraded":"2024-03-24T05:30:19.941013+0000","last_fullsized":"2024-03-24T05:30:19.941013+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:42.338022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'22","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939380+0000","last_change":"2024-03-24T05:29:53.698083+0000","last_active":"2024-03-24T05:30:19.939380+0000","last_peered":"2024-03-24T05:30:19.939380+0000","last_clean":"2024-03-24T05:30:19.939380+0000","last_became_active":"2024-03-24T05:29:53.697763+0000","last_became_peered":"2024-03-24T05:29:53.697763+0000","last_unstale":"2024-03-24T05:30:19.939380+0000","last_undegraded":"2024-03-24T05:30:19.939380+0000","last_fullsized":"2024-03-24T05:30:19.939380+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:47:29.474207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938199+0000","last_change":"2024-03-24T05:29:57.723770+0000","last_active":"2024-03-24T05:30:19.938199+0000","last_peered":"2024-03-24T05:30:19.938199+0000","last_clean":"2024-03-24T05:30:19.938199+0000","last_became_active":"2024-03-24T05:29:57.723278+0000","last_became_peered":"2024-03-24T05:29:57.723278+0000","last_unstale":"2024-03-24T05:30:19.938199+0000","last_undegraded":"2024-03-24T05:30:19.938199+0000","last_fullsized":"2024-03-24T05:30:19.938199+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:03:58.258463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938077+0000","last_change":"2024-03-24T05:29:55.715161+0000","last_active":"2024-03-24T05:30:19.938077+0000","last_peered":"2024-03-24T05:30:19.938077+0000","last_clean":"2024-03-24T05:30:19.938077+0000","last_became_active":"2024-03-24T05:29:55.714722+0000","last_became_peered":"2024-03-24T05:29:55.714722+0000","last_unstale":"2024-03-24T05:30:19.938077+0000","last_undegraded":"2024-03-24T05:30:19.938077+0000","last_fullsized":"2024-03-24T05:30:19.938077+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:54:34.389924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938865+0000","last_change":"2024-03-24T05:29:51.688513+0000","last_active":"2024-03-24T05:30:19.938865+0000","last_peered":"2024-03-24T05:30:19.938865+0000","last_clean":"2024-03-24T05:30:19.938865+0000","last_became_active":"2024-03-24T05:29:51.686135+0000","last_became_peered":"2024-03-24T05:29:51.686135+0000","last_unstale":"2024-03-24T05:30:19.938865+0000","last_undegraded":"2024-03-24T05:30:19.938865+0000","last_fullsized":"2024-03-24T05:30:19.938865+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:09:06.634054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940176+0000","last_change":"2024-03-24T05:29:53.698168+0000","last_active":"2024-03-24T05:30:19.940176+0000","last_peered":"2024-03-24T05:30:19.940176+0000","last_clean":"2024-03-24T05:30:19.940176+0000","last_became_active":"2024-03-24T05:29:53.697881+0000","last_became_peered":"2024-03-24T05:29:53.697881+0000","last_unstale":"2024-03-24T05:30:19.940176+0000","last_undegraded":"2024-03-24T05:30:19.940176+0000","last_fullsized":"2024-03-24T05:30:19.940176+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:29:49.365852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940134+0000","last_change":"2024-03-24T05:29:57.728360+0000","last_active":"2024-03-24T05:30:19.940134+0000","last_peered":"2024-03-24T05:30:19.940134+0000","last_clean":"2024-03-24T05:30:19.940134+0000","last_became_active":"2024-03-24T05:29:57.728052+0000","last_became_peered":"2024-03-24T05:29:57.728052+0000","last_unstale":"2024-03-24T05:30:19.940134+0000","last_undegraded":"2024-03-24T05:30:19.940134+0000","last_fullsized":"2024-03-24T05:30:19.940134+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:13:15.314499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:21.736207+0000","last_change":"2024-03-24T05:29:55.716925+0000","last_active":"2024-03-24T05:30:21.736207+0000","last_peered":"2024-03-24T05:30:21.736207+0000","last_clean":"2024-03-24T05:30:21.736207+0000","last_became_active":"2024-03-24T05:29:55.716516+0000","last_became_peered":"2024-03-24T05:29:55.716516+0000","last_unstale":"2024-03-24T05:30:21.736207+0000","last_undegraded":"2024-03-24T05:30:21.736207+0000","last_fullsized":"2024-03-24T05:30:21.736207+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:39.074860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938832+0000","last_change":"2024-03-24T05:29:51.690453+0000","last_active":"2024-03-24T05:30:19.938832+0000","last_peered":"2024-03-24T05:30:19.938832+0000","last_clean":"2024-03-24T05:30:19.938832+0000","last_became_active":"2024-03-24T05:29:51.689378+0000","last_became_peered":"2024-03-24T05:29:51.689378+0000","last_unstale":"2024-03-24T05:30:19.938832+0000","last_undegraded":"2024-03-24T05:30:19.938832+0000","last_fullsized":"2024-03-24T05:30:19.938832+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:35:50.258253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938225+0000","last_change":"2024-03-24T05:29:53.702889+0000","last_active":"2024-03-24T05:30:19.938225+0000","last_peered":"2024-03-24T05:30:19.938225+0000","last_clean":"2024-03-24T05:30:19.938225+0000","last_became_active":"2024-03-24T05:29:53.702386+0000","last_became_peered":"2024-03-24T05:29:53.702386+0000","last_unstale":"2024-03-24T05:30:19.938225+0000","last_undegraded":"2024-03-24T05:30:19.938225+0000","last_fullsized":"2024-03-24T05:30:19.938225+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:13.514023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938024+0000","last_change":"2024-03-24T05:29:57.726006+0000","last_active":"2024-03-24T05:30:19.938024+0000","last_peered":"2024-03-24T05:30:19.938024+0000","last_clean":"2024-03-24T05:30:19.938024+0000","last_became_active":"2024-03-24T05:29:57.725645+0000","last_became_peered":"2024-03-24T05:29:57.725645+0000","last_unstale":"2024-03-24T05:30:19.938024+0000","last_undegraded":"2024-03-24T05:30:19.938024+0000","last_fullsized":"2024-03-24T05:30:19.938024+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:36:01.547224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:21.736146+0000","last_change":"2024-03-24T05:29:55.712011+0000","last_active":"2024-03-24T05:30:21.736146+0000","last_peered":"2024-03-24T05:30:21.736146+0000","last_clean":"2024-03-24T05:30:21.736146+0000","last_became_active":"2024-03-24T05:29:55.711716+0000","last_became_peered":"2024-03-24T05:29:55.711716+0000","last_unstale":"2024-03-24T05:30:21.736146+0000","last_undegraded":"2024-03-24T05:30:21.736146+0000","last_fullsized":"2024-03-24T05:30:21.736146+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:31:18.147946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938331+0000","last_change":"2024-03-24T05:29:51.690262+0000","last_active":"2024-03-24T05:30:19.938331+0000","last_peered":"2024-03-24T05:30:19.938331+0000","last_clean":"2024-03-24T05:30:19.938331+0000","last_became_active":"2024-03-24T05:29:51.687739+0000","last_became_peered":"2024-03-24T05:29:51.687739+0000","last_unstale":"2024-03-24T05:30:19.938331+0000","last_undegraded":"2024-03-24T05:30:19.938331+0000","last_fullsized":"2024-03-24T05:30:19.938331+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:47.683861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938088+0000","last_change":"2024-03-24T05:29:53.697826+0000","last_active":"2024-03-24T05:30:19.938088+0000","last_peered":"2024-03-24T05:30:19.938088+0000","last_clean":"2024-03-24T05:30:19.938088+0000","last_became_active":"2024-03-24T05:29:53.697636+0000","last_became_peered":"2024-03-24T05:29:53.697636+0000","last_unstale":"2024-03-24T05:30:19.938088+0000","last_undegraded":"2024-03-24T05:30:19.938088+0000","last_fullsized":"2024-03-24T05:30:19.938088+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:38:19.391690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938204+0000","last_change":"2024-03-24T05:29:57.726588+0000","last_active":"2024-03-24T05:30:19.938204+0000","last_peered":"2024-03-24T05:30:19.938204+0000","last_clean":"2024-03-24T05:30:19.938204+0000","last_became_active":"2024-03-24T05:29:57.725791+0000","last_became_peered":"2024-03-24T05:29:57.725791+0000","last_unstale":"2024-03-24T05:30:19.938204+0000","last_undegraded":"2024-03-24T05:30:19.938204+0000","last_fullsized":"2024-03-24T05:30:19.938204+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:51:29.405480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940821+0000","last_change":"2024-03-24T05:29:55.719395+0000","last_active":"2024-03-24T05:30:19.940821+0000","last_peered":"2024-03-24T05:30:19.940821+0000","last_clean":"2024-03-24T05:30:19.940821+0000","last_became_active":"2024-03-24T05:29:55.718514+0000","last_became_peered":"2024-03-24T05:29:55.718514+0000","last_unstale":"2024-03-24T05:30:19.940821+0000","last_undegraded":"2024-03-24T05:30:19.940821+0000","last_fullsized":"2024-03-24T05:30:19.940821+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:10.861395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937064+0000","last_change":"2024-03-24T05:29:51.685235+0000","last_active":"2024-03-24T05:30:19.937064+0000","last_peered":"2024-03-24T05:30:19.937064+0000","last_clean":"2024-03-24T05:30:19.937064+0000","last_became_active":"2024-03-24T05:29:51.684606+0000","last_became_peered":"2024-03-24T05:29:51.684606+0000","last_unstale":"2024-03-24T05:30:19.937064+0000","last_undegraded":"2024-03-24T05:30:19.937064+0000","last_fullsized":"2024-03-24T05:30:19.937064+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:06:10.422744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.936758+0000","last_change":"2024-03-24T05:29:53.697197+0000","last_active":"2024-03-24T05:30:19.936758+0000","last_peered":"2024-03-24T05:30:19.936758+0000","last_clean":"2024-03-24T05:30:19.936758+0000","last_became_active":"2024-03-24T05:29:53.696835+0000","last_became_peered":"2024-03-24T05:29:53.696835+0000","last_unstale":"2024-03-24T05:30:19.936758+0000","last_undegraded":"2024-03-24T05:30:19.936758+0000","last_fullsized":"2024-03-24T05:30:19.936758+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:17:13.302550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940536+0000","last_change":"2024-03-24T05:29:57.729138+0000","last_active":"2024-03-24T05:30:19.940536+0000","last_peered":"2024-03-24T05:30:19.940536+0000","last_clean":"2024-03-24T05:30:19.940536+0000","last_became_active":"2024-03-24T05:29:57.728830+0000","last_became_peered":"2024-03-24T05:29:57.728830+0000","last_unstale":"2024-03-24T05:30:19.940536+0000","last_undegraded":"2024-03-24T05:30:19.940536+0000","last_fullsized":"2024-03-24T05:30:19.940536+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:42:16.818621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941792+0000","last_change":"2024-03-24T05:29:55.715995+0000","last_active":"2024-03-24T05:30:19.941792+0000","last_peered":"2024-03-24T05:30:19.941792+0000","last_clean":"2024-03-24T05:30:19.941792+0000","last_became_active":"2024-03-24T05:29:55.715673+0000","last_became_peered":"2024-03-24T05:29:55.715673+0000","last_unstale":"2024-03-24T05:30:19.941792+0000","last_undegraded":"2024-03-24T05:30:19.941792+0000","last_fullsized":"2024-03-24T05:30:19.941792+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:08:13.673922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938548+0000","last_change":"2024-03-24T05:29:51.690930+0000","last_active":"2024-03-24T05:30:19.938548+0000","last_peered":"2024-03-24T05:30:19.938548+0000","last_clean":"2024-03-24T05:30:19.938548+0000","last_became_active":"2024-03-24T05:29:51.688169+0000","last_became_peered":"2024-03-24T05:29:51.688169+0000","last_unstale":"2024-03-24T05:30:19.938548+0000","last_undegraded":"2024-03-24T05:30:19.938548+0000","last_fullsized":"2024-03-24T05:30:19.938548+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:12:16.953006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939501+0000","last_change":"2024-03-24T05:29:53.697410+0000","last_active":"2024-03-24T05:30:19.939501+0000","last_peered":"2024-03-24T05:30:19.939501+0000","last_clean":"2024-03-24T05:30:19.939501+0000","last_became_active":"2024-03-24T05:29:53.696603+0000","last_became_peered":"2024-03-24T05:29:53.696603+0000","last_unstale":"2024-03-24T05:30:19.939501+0000","last_undegraded":"2024-03-24T05:30:19.939501+0000","last_fullsized":"2024-03-24T05:30:19.939501+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:15:23.001096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939212+0000","last_change":"2024-03-24T05:29:57.722171+0000","last_active":"2024-03-24T05:30:19.939212+0000","last_peered":"2024-03-24T05:30:19.939212+0000","last_clean":"2024-03-24T05:30:19.939212+0000","last_became_active":"2024-03-24T05:29:57.721917+0000","last_became_peered":"2024-03-24T05:29:57.721917+0000","last_unstale":"2024-03-24T05:30:19.939212+0000","last_undegraded":"2024-03-24T05:30:19.939212+0000","last_fullsized":"2024-03-24T05:30:19.939212+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:27:37.865157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939778+0000","last_change":"2024-03-24T05:29:51.688269+0000","last_active":"2024-03-24T05:30:19.939778+0000","last_peered":"2024-03-24T05:30:19.939778+0000","last_clean":"2024-03-24T05:30:19.939778+0000","last_became_active":"2024-03-24T05:29:51.685514+0000","last_became_peered":"2024-03-24T05:29:51.685514+0000","last_unstale":"2024-03-24T05:30:19.939778+0000","last_undegraded":"2024-03-24T05:30:19.939778+0000","last_fullsized":"2024-03-24T05:30:19.939778+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:01.333873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940994+0000","last_change":"2024-03-24T05:29:53.700911+0000","last_active":"2024-03-24T05:30:19.940994+0000","last_peered":"2024-03-24T05:30:19.940994+0000","last_clean":"2024-03-24T05:30:19.940994+0000","last_became_active":"2024-03-24T05:29:53.700610+0000","last_became_peered":"2024-03-24T05:29:53.700610+0000","last_unstale":"2024-03-24T05:30:19.940994+0000","last_undegraded":"2024-03-24T05:30:19.940994+0000","last_fullsized":"2024-03-24T05:30:19.940994+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:43:18.485486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:21.736181+0000","last_change":"2024-03-24T05:29:55.716102+0000","last_active":"2024-03-24T05:30:21.736181+0000","last_peered":"2024-03-24T05:30:21.736181+0000","last_clean":"2024-03-24T05:30:21.736181+0000","last_became_active":"2024-03-24T05:29:55.715754+0000","last_became_peered":"2024-03-24T05:29:55.715754+0000","last_unstale":"2024-03-24T05:30:21.736181+0000","last_undegraded":"2024-03-24T05:30:21.736181+0000","last_fullsized":"2024-03-24T05:30:21.736181+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:02.819322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938332+0000","last_change":"2024-03-24T05:29:57.726672+0000","last_active":"2024-03-24T05:30:19.938332+0000","last_peered":"2024-03-24T05:30:19.938332+0000","last_clean":"2024-03-24T05:30:19.938332+0000","last_became_active":"2024-03-24T05:29:57.726320+0000","last_became_peered":"2024-03-24T05:29:57.726320+0000","last_unstale":"2024-03-24T05:30:19.938332+0000","last_undegraded":"2024-03-24T05:30:19.938332+0000","last_fullsized":"2024-03-24T05:30:19.938332+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:14:33.520133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940723+0000","last_change":"2024-03-24T05:29:51.691807+0000","last_active":"2024-03-24T05:30:19.940723+0000","last_peered":"2024-03-24T05:30:19.940723+0000","last_clean":"2024-03-24T05:30:19.940723+0000","last_became_active":"2024-03-24T05:29:51.691273+0000","last_became_peered":"2024-03-24T05:29:51.691273+0000","last_unstale":"2024-03-24T05:30:19.940723+0000","last_undegraded":"2024-03-24T05:30:19.940723+0000","last_fullsized":"2024-03-24T05:30:19.940723+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:51.744600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:21.735980+0000","last_change":"2024-03-24T05:29:53.697131+0000","last_active":"2024-03-24T05:30:21.735980+0000","last_peered":"2024-03-24T05:30:21.735980+0000","last_clean":"2024-03-24T05:30:21.735980+0000","last_became_active":"2024-03-24T05:29:53.696772+0000","last_became_peered":"2024-03-24T05:29:53.696772+0000","last_unstale":"2024-03-24T05:30:21.735980+0000","last_undegraded":"2024-03-24T05:30:21.735980+0000","last_fullsized":"2024-03-24T05:30:21.735980+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:01:16.901253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938203+0000","last_change":"2024-03-24T05:29:55.716629+0000","last_active":"2024-03-24T05:30:19.938203+0000","last_peered":"2024-03-24T05:30:19.938203+0000","last_clean":"2024-03-24T05:30:19.938203+0000","last_became_active":"2024-03-24T05:29:55.716266+0000","last_became_peered":"2024-03-24T05:29:55.716266+0000","last_unstale":"2024-03-24T05:30:19.938203+0000","last_undegraded":"2024-03-24T05:30:19.938203+0000","last_fullsized":"2024-03-24T05:30:19.938203+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:24:59.772747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939389+0000","last_change":"2024-03-24T05:29:57.727150+0000","last_active":"2024-03-24T05:30:19.939389+0000","last_peered":"2024-03-24T05:30:19.939389+0000","last_clean":"2024-03-24T05:30:19.939389+0000","last_became_active":"2024-03-24T05:29:57.726853+0000","last_became_peered":"2024-03-24T05:29:57.726853+0000","last_unstale":"2024-03-24T05:30:19.939389+0000","last_undegraded":"2024-03-24T05:30:19.939389+0000","last_fullsized":"2024-03-24T05:30:19.939389+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:30:44.744538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937459+0000","last_change":"2024-03-24T05:29:51.687266+0000","last_active":"2024-03-24T05:30:19.937459+0000","last_peered":"2024-03-24T05:30:19.937459+0000","last_clean":"2024-03-24T05:30:19.937459+0000","last_became_active":"2024-03-24T05:29:51.686218+0000","last_became_peered":"2024-03-24T05:29:51.686218+0000","last_unstale":"2024-03-24T05:30:19.937459+0000","last_undegraded":"2024-03-24T05:30:19.937459+0000","last_fullsized":"2024-03-24T05:30:19.937459+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:47:06.136824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'25","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938289+0000","last_change":"2024-03-24T05:29:53.697423+0000","last_active":"2024-03-24T05:30:19.938289+0000","last_peered":"2024-03-24T05:30:19.938289+0000","last_clean":"2024-03-24T05:30:19.938289+0000","last_became_active":"2024-03-24T05:29:53.696958+0000","last_became_peered":"2024-03-24T05:29:53.696958+0000","last_unstale":"2024-03-24T05:30:19.938289+0000","last_undegraded":"2024-03-24T05:30:19.938289+0000","last_fullsized":"2024-03-24T05:30:19.938289+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:46:11.436240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940235+0000","last_change":"2024-03-24T05:29:55.715462+0000","last_active":"2024-03-24T05:30:19.940235+0000","last_peered":"2024-03-24T05:30:19.940235+0000","last_clean":"2024-03-24T05:30:19.940235+0000","last_became_active":"2024-03-24T05:29:55.714892+0000","last_became_peered":"2024-03-24T05:29:55.714892+0000","last_unstale":"2024-03-24T05:30:19.940235+0000","last_undegraded":"2024-03-24T05:30:19.940235+0000","last_fullsized":"2024-03-24T05:30:19.940235+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:38:04.257745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941906+0000","last_change":"2024-03-24T05:29:57.728305+0000","last_active":"2024-03-24T05:30:19.941906+0000","last_peered":"2024-03-24T05:30:19.941906+0000","last_clean":"2024-03-24T05:30:19.941906+0000","last_became_active":"2024-03-24T05:29:57.727814+0000","last_became_peered":"2024-03-24T05:29:57.727814+0000","last_unstale":"2024-03-24T05:30:19.941906+0000","last_undegraded":"2024-03-24T05:30:19.941906+0000","last_fullsized":"2024-03-24T05:30:19.941906+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:54:26.778782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'30","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937897+0000","last_change":"2024-03-24T05:29:53.697394+0000","last_active":"2024-03-24T05:30:19.937897+0000","last_peered":"2024-03-24T05:30:19.937897+0000","last_clean":"2024-03-24T05:30:19.937897+0000","last_became_active":"2024-03-24T05:29:53.696673+0000","last_became_peered":"2024-03-24T05:29:53.696673+0000","last_unstale":"2024-03-24T05:30:19.937897+0000","last_undegraded":"2024-03-24T05:30:19.937897+0000","last_fullsized":"2024-03-24T05:30:19.937897+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:31:07.573292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938015+0000","last_change":"2024-03-24T05:29:51.691707+0000","last_active":"2024-03-24T05:30:19.938015+0000","last_peered":"2024-03-24T05:30:19.938015+0000","last_clean":"2024-03-24T05:30:19.938015+0000","last_became_active":"2024-03-24T05:29:51.691434+0000","last_became_peered":"2024-03-24T05:29:51.691434+0000","last_unstale":"2024-03-24T05:30:19.938015+0000","last_undegraded":"2024-03-24T05:30:19.938015+0000","last_fullsized":"2024-03-24T05:30:19.938015+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:30.571121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941137+0000","last_change":"2024-03-24T05:29:55.719677+0000","last_active":"2024-03-24T05:30:19.941137+0000","last_peered":"2024-03-24T05:30:19.941137+0000","last_clean":"2024-03-24T05:30:19.941137+0000","last_became_active":"2024-03-24T05:29:55.719074+0000","last_became_peered":"2024-03-24T05:29:55.719074+0000","last_unstale":"2024-03-24T05:30:19.941137+0000","last_undegraded":"2024-03-24T05:30:19.941137+0000","last_fullsized":"2024-03-24T05:30:19.941137+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:09:58.501606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938431+0000","last_change":"2024-03-24T05:29:57.723856+0000","last_active":"2024-03-24T05:30:19.938431+0000","last_peered":"2024-03-24T05:30:19.938431+0000","last_clean":"2024-03-24T05:30:19.938431+0000","last_became_active":"2024-03-24T05:29:57.723213+0000","last_became_peered":"2024-03-24T05:29:57.723213+0000","last_unstale":"2024-03-24T05:30:19.938431+0000","last_undegraded":"2024-03-24T05:30:19.938431+0000","last_fullsized":"2024-03-24T05:30:19.938431+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:52:56.570403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938153+0000","last_change":"2024-03-24T05:29:53.702621+0000","last_active":"2024-03-24T05:30:19.938153+0000","last_peered":"2024-03-24T05:30:19.938153+0000","last_clean":"2024-03-24T05:30:19.938153+0000","last_became_active":"2024-03-24T05:29:53.701835+0000","last_became_peered":"2024-03-24T05:29:53.701835+0000","last_unstale":"2024-03-24T05:30:19.938153+0000","last_undegraded":"2024-03-24T05:30:19.938153+0000","last_fullsized":"2024-03-24T05:30:19.938153+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:04:55.490735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937908+0000","last_change":"2024-03-24T05:29:51.691812+0000","last_active":"2024-03-24T05:30:19.937908+0000","last_peered":"2024-03-24T05:30:19.937908+0000","last_clean":"2024-03-24T05:30:19.937908+0000","last_became_active":"2024-03-24T05:29:51.691360+0000","last_became_peered":"2024-03-24T05:29:51.691360+0000","last_unstale":"2024-03-24T05:30:19.937908+0000","last_undegraded":"2024-03-24T05:30:19.937908+0000","last_fullsized":"2024-03-24T05:30:19.937908+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:28:31.405765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:22.040617+0000","last_change":"2024-03-24T05:29:37.691758+0000","last_active":"2024-03-24T05:30:22.040617+0000","last_peered":"2024-03-24T05:30:22.040617+0000","last_clean":"2024-03-24T05:30:22.040617+0000","last_became_active":"2024-03-24T05:29:37.691498+0000","last_became_peered":"2024-03-24T05:29:37.691498+0000","last_unstale":"2024-03-24T05:30:22.040617+0000","last_undegraded":"2024-03-24T05:30:22.040617+0000","last_fullsized":"2024-03-24T05:30:22.040617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:36.682205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:36.682205+0000","last_clean_scrub_stamp":"2024-03-24T05:29:36.682205+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:16:56.278660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938529+0000","last_change":"2024-03-24T05:29:55.715197+0000","last_active":"2024-03-24T05:30:19.938529+0000","last_peered":"2024-03-24T05:30:19.938529+0000","last_clean":"2024-03-24T05:30:19.938529+0000","last_became_active":"2024-03-24T05:29:55.714317+0000","last_became_peered":"2024-03-24T05:29:55.714317+0000","last_unstale":"2024-03-24T05:30:19.938529+0000","last_undegraded":"2024-03-24T05:30:19.938529+0000","last_fullsized":"2024-03-24T05:30:19.938529+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:50:16.391248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938764+0000","last_change":"2024-03-24T05:29:57.724944+0000","last_active":"2024-03-24T05:30:19.938764+0000","last_peered":"2024-03-24T05:30:19.938764+0000","last_clean":"2024-03-24T05:30:19.938764+0000","last_became_active":"2024-03-24T05:29:57.723662+0000","last_became_peered":"2024-03-24T05:29:57.723662+0000","last_unstale":"2024-03-24T05:30:19.938764+0000","last_undegraded":"2024-03-24T05:30:19.938764+0000","last_fullsized":"2024-03-24T05:30:19.938764+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:00:28.061671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'20","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938219+0000","last_change":"2024-03-24T05:29:53.703755+0000","last_active":"2024-03-24T05:30:19.938219+0000","last_peered":"2024-03-24T05:30:19.938219+0000","last_clean":"2024-03-24T05:30:19.938219+0000","last_became_active":"2024-03-24T05:29:53.703156+0000","last_became_peered":"2024-03-24T05:29:53.703156+0000","last_unstale":"2024-03-24T05:30:19.938219+0000","last_undegraded":"2024-03-24T05:30:19.938219+0000","last_fullsized":"2024-03-24T05:30:19.938219+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:44.462349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.936842+0000","last_change":"2024-03-24T05:29:51.687201+0000","last_active":"2024-03-24T05:30:19.936842+0000","last_peered":"2024-03-24T05:30:19.936842+0000","last_clean":"2024-03-24T05:30:19.936842+0000","last_became_active":"2024-03-24T05:29:51.686074+0000","last_became_peered":"2024-03-24T05:29:51.686074+0000","last_unstale":"2024-03-24T05:30:19.936842+0000","last_undegraded":"2024-03-24T05:30:19.936842+0000","last_fullsized":"2024-03-24T05:30:19.936842+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:56:57.331519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940073+0000","last_change":"2024-03-24T05:29:55.713355+0000","last_active":"2024-03-24T05:30:19.940073+0000","last_peered":"2024-03-24T05:30:19.940073+0000","last_clean":"2024-03-24T05:30:19.940073+0000","last_became_active":"2024-03-24T05:29:55.712999+0000","last_became_peered":"2024-03-24T05:29:55.712999+0000","last_unstale":"2024-03-24T05:30:19.940073+0000","last_undegraded":"2024-03-24T05:30:19.940073+0000","last_fullsized":"2024-03-24T05:30:19.940073+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:44:29.618389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937336+0000","last_change":"2024-03-24T05:29:57.724557+0000","last_active":"2024-03-24T05:30:19.937336+0000","last_peered":"2024-03-24T05:30:19.937336+0000","last_clean":"2024-03-24T05:30:19.937336+0000","last_became_active":"2024-03-24T05:29:57.723634+0000","last_became_peered":"2024-03-24T05:29:57.723634+0000","last_unstale":"2024-03-24T05:30:19.937336+0000","last_undegraded":"2024-03-24T05:30:19.937336+0000","last_fullsized":"2024-03-24T05:30:19.937336+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:10.119301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940942+0000","last_change":"2024-03-24T05:29:51.691613+0000","last_active":"2024-03-24T05:30:19.940942+0000","last_peered":"2024-03-24T05:30:19.940942+0000","last_clean":"2024-03-24T05:30:19.940942+0000","last_became_active":"2024-03-24T05:29:51.690522+0000","last_became_peered":"2024-03-24T05:29:51.690522+0000","last_unstale":"2024-03-24T05:30:19.940942+0000","last_undegraded":"2024-03-24T05:30:19.940942+0000","last_fullsized":"2024-03-24T05:30:19.940942+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:14:17.009626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'36","reported_seq":93,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:21.736086+0000","last_change":"2024-03-24T05:29:53.703629+0000","last_active":"2024-03-24T05:30:21.736086+0000","last_peered":"2024-03-24T05:30:21.736086+0000","last_clean":"2024-03-24T05:30:21.736086+0000","last_became_active":"2024-03-24T05:29:53.703156+0000","last_became_peered":"2024-03-24T05:29:53.703156+0000","last_unstale":"2024-03-24T05:30:21.736086+0000","last_undegraded":"2024-03-24T05:30:21.736086+0000","last_fullsized":"2024-03-24T05:30:21.736086+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:09.753001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:21.736175+0000","last_change":"2024-03-24T05:29:55.713003+0000","last_active":"2024-03-24T05:30:21.736175+0000","last_peered":"2024-03-24T05:30:21.736175+0000","last_clean":"2024-03-24T05:30:21.736175+0000","last_became_active":"2024-03-24T05:29:55.712651+0000","last_became_peered":"2024-03-24T05:29:55.712651+0000","last_unstale":"2024-03-24T05:30:21.736175+0000","last_undegraded":"2024-03-24T05:30:21.736175+0000","last_fullsized":"2024-03-24T05:30:21.736175+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:19:05.588179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.936854+0000","last_change":"2024-03-24T05:29:57.723905+0000","last_active":"2024-03-24T05:30:19.936854+0000","last_peered":"2024-03-24T05:30:19.936854+0000","last_clean":"2024-03-24T05:30:19.936854+0000","last_became_active":"2024-03-24T05:29:57.723296+0000","last_became_peered":"2024-03-24T05:29:57.723296+0000","last_unstale":"2024-03-24T05:30:19.936854+0000","last_undegraded":"2024-03-24T05:30:19.936854+0000","last_fullsized":"2024-03-24T05:30:19.936854+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:36.494956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.942243+0000","last_change":"2024-03-24T05:29:51.692615+0000","last_active":"2024-03-24T05:30:19.942243+0000","last_peered":"2024-03-24T05:30:19.942243+0000","last_clean":"2024-03-24T05:30:19.942243+0000","last_became_active":"2024-03-24T05:29:51.692164+0000","last_became_peered":"2024-03-24T05:29:51.692164+0000","last_unstale":"2024-03-24T05:30:19.942243+0000","last_undegraded":"2024-03-24T05:30:19.942243+0000","last_fullsized":"2024-03-24T05:30:19.942243+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:52.830091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'19","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941687+0000","last_change":"2024-03-24T05:29:53.704033+0000","last_active":"2024-03-24T05:30:19.941687+0000","last_peered":"2024-03-24T05:30:19.941687+0000","last_clean":"2024-03-24T05:30:19.941687+0000","last_became_active":"2024-03-24T05:29:53.703741+0000","last_became_peered":"2024-03-24T05:29:53.703741+0000","last_unstale":"2024-03-24T05:30:19.941687+0000","last_undegraded":"2024-03-24T05:30:19.941687+0000","last_fullsized":"2024-03-24T05:30:19.941687+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:59:19.419593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:21.736060+0000","last_change":"2024-03-24T05:29:55.715604+0000","last_active":"2024-03-24T05:30:21.736060+0000","last_peered":"2024-03-24T05:30:21.736060+0000","last_clean":"2024-03-24T05:30:21.736060+0000","last_became_active":"2024-03-24T05:29:55.714909+0000","last_became_peered":"2024-03-24T05:29:55.714909+0000","last_unstale":"2024-03-24T05:30:21.736060+0000","last_undegraded":"2024-03-24T05:30:21.736060+0000","last_fullsized":"2024-03-24T05:30:21.736060+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:28:22.930231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939057+0000","last_change":"2024-03-24T05:29:57.722194+0000","last_active":"2024-03-24T05:30:19.939057+0000","last_peered":"2024-03-24T05:30:19.939057+0000","last_clean":"2024-03-24T05:30:19.939057+0000","last_became_active":"2024-03-24T05:29:57.721898+0000","last_became_peered":"2024-03-24T05:29:57.721898+0000","last_unstale":"2024-03-24T05:30:19.939057+0000","last_undegraded":"2024-03-24T05:30:19.939057+0000","last_fullsized":"2024-03-24T05:30:19.939057+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:58:27.261499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939310+0000","last_change":"2024-03-24T05:29:51.687572+0000","last_active":"2024-03-24T05:30:19.939310+0000","last_peered":"2024-03-24T05:30:19.939310+0000","last_clean":"2024-03-24T05:30:19.939310+0000","last_became_active":"2024-03-24T05:29:51.686354+0000","last_became_peered":"2024-03-24T05:29:51.686354+0000","last_unstale":"2024-03-24T05:30:19.939310+0000","last_undegraded":"2024-03-24T05:30:19.939310+0000","last_fullsized":"2024-03-24T05:30:19.939310+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:44:29.754475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941882+0000","last_change":"2024-03-24T05:29:53.705417+0000","last_active":"2024-03-24T05:30:19.941882+0000","last_peered":"2024-03-24T05:30:19.941882+0000","last_clean":"2024-03-24T05:30:19.941882+0000","last_became_active":"2024-03-24T05:29:53.705071+0000","last_became_peered":"2024-03-24T05:29:53.705071+0000","last_unstale":"2024-03-24T05:30:19.941882+0000","last_undegraded":"2024-03-24T05:30:19.941882+0000","last_fullsized":"2024-03-24T05:30:19.941882+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:57:15.142142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938173+0000","last_change":"2024-03-24T05:29:55.713051+0000","last_active":"2024-03-24T05:30:19.938173+0000","last_peered":"2024-03-24T05:30:19.938173+0000","last_clean":"2024-03-24T05:30:19.938173+0000","last_became_active":"2024-03-24T05:29:55.712717+0000","last_became_peered":"2024-03-24T05:29:55.712717+0000","last_unstale":"2024-03-24T05:30:19.938173+0000","last_undegraded":"2024-03-24T05:30:19.938173+0000","last_fullsized":"2024-03-24T05:30:19.938173+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:48:15.546019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938860+0000","last_change":"2024-03-24T05:29:57.726760+0000","last_active":"2024-03-24T05:30:19.938860+0000","last_peered":"2024-03-24T05:30:19.938860+0000","last_clean":"2024-03-24T05:30:19.938860+0000","last_became_active":"2024-03-24T05:29:57.725013+0000","last_became_peered":"2024-03-24T05:29:57.725013+0000","last_unstale":"2024-03-24T05:30:19.938860+0000","last_undegraded":"2024-03-24T05:30:19.938860+0000","last_fullsized":"2024-03-24T05:30:19.938860+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:34:04.413542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:21.735930+0000","last_change":"2024-03-24T05:29:55.716980+0000","last_active":"2024-03-24T05:30:21.735930+0000","last_peered":"2024-03-24T05:30:21.735930+0000","last_clean":"2024-03-24T05:30:21.735930+0000","last_became_active":"2024-03-24T05:29:55.715918+0000","last_became_peered":"2024-03-24T05:29:55.715918+0000","last_unstale":"2024-03-24T05:30:21.735930+0000","last_undegraded":"2024-03-24T05:30:21.735930+0000","last_fullsized":"2024-03-24T05:30:21.735930+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:30.346542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.936941+0000","last_change":"2024-03-24T05:29:51.688420+0000","last_active":"2024-03-24T05:30:19.936941+0000","last_peered":"2024-03-24T05:30:19.936941+0000","last_clean":"2024-03-24T05:30:19.936941+0000","last_became_active":"2024-03-24T05:29:51.686381+0000","last_became_peered":"2024-03-24T05:29:51.686381+0000","last_unstale":"2024-03-24T05:30:19.936941+0000","last_undegraded":"2024-03-24T05:30:19.936941+0000","last_fullsized":"2024-03-24T05:30:19.936941+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:35.244318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'24","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939719+0000","last_change":"2024-03-24T05:29:53.698224+0000","last_active":"2024-03-24T05:30:19.939719+0000","last_peered":"2024-03-24T05:30:19.939719+0000","last_clean":"2024-03-24T05:30:19.939719+0000","last_became_active":"2024-03-24T05:29:53.697700+0000","last_became_peered":"2024-03-24T05:29:53.697700+0000","last_unstale":"2024-03-24T05:30:19.939719+0000","last_undegraded":"2024-03-24T05:30:19.939719+0000","last_fullsized":"2024-03-24T05:30:19.939719+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:40:05.984990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.941142+0000","last_change":"2024-03-24T05:29:57.724588+0000","last_active":"2024-03-24T05:30:19.941142+0000","last_peered":"2024-03-24T05:30:19.941142+0000","last_clean":"2024-03-24T05:30:19.941142+0000","last_became_active":"2024-03-24T05:29:57.723703+0000","last_became_peered":"2024-03-24T05:29:57.723703+0000","last_unstale":"2024-03-24T05:30:19.941142+0000","last_undegraded":"2024-03-24T05:30:19.941142+0000","last_fullsized":"2024-03-24T05:30:19.941142+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:47:54.467370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940912+0000","last_change":"2024-03-24T05:29:55.719458+0000","last_active":"2024-03-24T05:30:19.940912+0000","last_peered":"2024-03-24T05:30:19.940912+0000","last_clean":"2024-03-24T05:30:19.940912+0000","last_became_active":"2024-03-24T05:29:55.718893+0000","last_became_peered":"2024-03-24T05:29:55.718893+0000","last_unstale":"2024-03-24T05:30:19.940912+0000","last_undegraded":"2024-03-24T05:30:19.940912+0000","last_fullsized":"2024-03-24T05:30:19.940912+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:14:43.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938468+0000","last_change":"2024-03-24T05:29:51.691184+0000","last_active":"2024-03-24T05:30:19.938468+0000","last_peered":"2024-03-24T05:30:19.938468+0000","last_clean":"2024-03-24T05:30:19.938468+0000","last_became_active":"2024-03-24T05:29:51.690900+0000","last_became_peered":"2024-03-24T05:29:51.690900+0000","last_unstale":"2024-03-24T05:30:19.938468+0000","last_undegraded":"2024-03-24T05:30:19.938468+0000","last_fullsized":"2024-03-24T05:30:19.938468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:30:04.668568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'27","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938642+0000","last_change":"2024-03-24T05:29:53.697237+0000","last_active":"2024-03-24T05:30:19.938642+0000","last_peered":"2024-03-24T05:30:19.938642+0000","last_clean":"2024-03-24T05:30:19.938642+0000","last_became_active":"2024-03-24T05:29:53.696612+0000","last_became_peered":"2024-03-24T05:29:53.696612+0000","last_unstale":"2024-03-24T05:30:19.938642+0000","last_undegraded":"2024-03-24T05:30:19.938642+0000","last_fullsized":"2024-03-24T05:30:19.938642+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:03:52.698037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939922+0000","last_change":"2024-03-24T05:29:57.724736+0000","last_active":"2024-03-24T05:30:19.939922+0000","last_peered":"2024-03-24T05:30:19.939922+0000","last_clean":"2024-03-24T05:30:19.939922+0000","last_became_active":"2024-03-24T05:29:57.723857+0000","last_became_peered":"2024-03-24T05:29:57.723857+0000","last_unstale":"2024-03-24T05:30:19.939922+0000","last_undegraded":"2024-03-24T05:30:19.939922+0000","last_fullsized":"2024-03-24T05:30:19.939922+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:52:43.908548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938740+0000","last_change":"2024-03-24T05:29:51.687685+0000","last_active":"2024-03-24T05:30:19.938740+0000","last_peered":"2024-03-24T05:30:19.938740+0000","last_clean":"2024-03-24T05:30:19.938740+0000","last_became_active":"2024-03-24T05:29:51.686005+0000","last_became_peered":"2024-03-24T05:29:51.686005+0000","last_unstale":"2024-03-24T05:30:19.938740+0000","last_undegraded":"2024-03-24T05:30:19.938740+0000","last_fullsized":"2024-03-24T05:30:19.938740+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:45:52.923151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938637+0000","last_change":"2024-03-24T05:29:53.703787+0000","last_active":"2024-03-24T05:30:19.938637+0000","last_peered":"2024-03-24T05:30:19.938637+0000","last_clean":"2024-03-24T05:30:19.938637+0000","last_became_active":"2024-03-24T05:29:53.703332+0000","last_became_peered":"2024-03-24T05:29:53.703332+0000","last_unstale":"2024-03-24T05:30:19.938637+0000","last_undegraded":"2024-03-24T05:30:19.938637+0000","last_fullsized":"2024-03-24T05:30:19.938637+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:31.111846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938507+0000","last_change":"2024-03-24T05:29:55.716849+0000","last_active":"2024-03-24T05:30:19.938507+0000","last_peered":"2024-03-24T05:30:19.938507+0000","last_clean":"2024-03-24T05:30:19.938507+0000","last_became_active":"2024-03-24T05:29:55.715768+0000","last_became_peered":"2024-03-24T05:29:55.715768+0000","last_unstale":"2024-03-24T05:30:19.938507+0000","last_undegraded":"2024-03-24T05:30:19.938507+0000","last_fullsized":"2024-03-24T05:30:19.938507+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:48:05.840788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939858+0000","last_change":"2024-03-24T05:29:57.728423+0000","last_active":"2024-03-24T05:30:19.939858+0000","last_peered":"2024-03-24T05:30:19.939858+0000","last_clean":"2024-03-24T05:30:19.939858+0000","last_became_active":"2024-03-24T05:29:57.728106+0000","last_became_peered":"2024-03-24T05:29:57.728106+0000","last_unstale":"2024-03-24T05:30:19.939858+0000","last_undegraded":"2024-03-24T05:30:19.939858+0000","last_fullsized":"2024-03-24T05:30:19.939858+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:12:36.838301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:21.735945+0000","last_change":"2024-03-24T05:29:55.715615+0000","last_active":"2024-03-24T05:30:21.735945+0000","last_peered":"2024-03-24T05:30:21.735945+0000","last_clean":"2024-03-24T05:30:21.735945+0000","last_became_active":"2024-03-24T05:29:55.715210+0000","last_became_peered":"2024-03-24T05:29:55.715210+0000","last_unstale":"2024-03-24T05:30:21.735945+0000","last_undegraded":"2024-03-24T05:30:21.735945+0000","last_fullsized":"2024-03-24T05:30:21.735945+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:29:07.973624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939195+0000","last_change":"2024-03-24T05:29:51.688997+0000","last_active":"2024-03-24T05:30:19.939195+0000","last_peered":"2024-03-24T05:30:19.939195+0000","last_clean":"2024-03-24T05:30:19.939195+0000","last_became_active":"2024-03-24T05:29:51.688525+0000","last_became_peered":"2024-03-24T05:29:51.688525+0000","last_unstale":"2024-03-24T05:30:19.939195+0000","last_undegraded":"2024-03-24T05:30:19.939195+0000","last_fullsized":"2024-03-24T05:30:19.939195+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:26:08.514377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'25","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937550+0000","last_change":"2024-03-24T05:29:53.697062+0000","last_active":"2024-03-24T05:30:19.937550+0000","last_peered":"2024-03-24T05:30:19.937550+0000","last_clean":"2024-03-24T05:30:19.937550+0000","last_became_active":"2024-03-24T05:29:53.696765+0000","last_became_peered":"2024-03-24T05:29:53.696765+0000","last_unstale":"2024-03-24T05:30:19.937550+0000","last_undegraded":"2024-03-24T05:30:19.937550+0000","last_fullsized":"2024-03-24T05:30:19.937550+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:11:36.773653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938422+0000","last_change":"2024-03-24T05:29:57.724736+0000","last_active":"2024-03-24T05:30:19.938422+0000","last_peered":"2024-03-24T05:30:19.938422+0000","last_clean":"2024-03-24T05:30:19.938422+0000","last_became_active":"2024-03-24T05:29:57.723840+0000","last_became_peered":"2024-03-24T05:29:57.723840+0000","last_unstale":"2024-03-24T05:30:19.938422+0000","last_undegraded":"2024-03-24T05:30:19.938422+0000","last_fullsized":"2024-03-24T05:30:19.938422+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:02:24.814339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939597+0000","last_change":"2024-03-24T05:29:51.687476+0000","last_active":"2024-03-24T05:30:19.939597+0000","last_peered":"2024-03-24T05:30:19.939597+0000","last_clean":"2024-03-24T05:30:19.939597+0000","last_became_active":"2024-03-24T05:29:51.686500+0000","last_became_peered":"2024-03-24T05:29:51.686500+0000","last_unstale":"2024-03-24T05:30:19.939597+0000","last_undegraded":"2024-03-24T05:30:19.939597+0000","last_fullsized":"2024-03-24T05:30:19.939597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:50:25.773727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938961+0000","last_change":"2024-03-24T05:29:53.697001+0000","last_active":"2024-03-24T05:30:19.938961+0000","last_peered":"2024-03-24T05:30:19.938961+0000","last_clean":"2024-03-24T05:30:19.938961+0000","last_became_active":"2024-03-24T05:29:53.696653+0000","last_became_peered":"2024-03-24T05:29:53.696653+0000","last_unstale":"2024-03-24T05:30:19.938961+0000","last_undegraded":"2024-03-24T05:30:19.938961+0000","last_fullsized":"2024-03-24T05:30:19.938961+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:09:48.897812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937238+0000","last_change":"2024-03-24T05:29:55.715421+0000","last_active":"2024-03-24T05:30:19.937238+0000","last_peered":"2024-03-24T05:30:19.937238+0000","last_clean":"2024-03-24T05:30:19.937238+0000","last_became_active":"2024-03-24T05:29:55.714893+0000","last_became_peered":"2024-03-24T05:29:55.714893+0000","last_unstale":"2024-03-24T05:30:19.937238+0000","last_undegraded":"2024-03-24T05:30:19.937238+0000","last_fullsized":"2024-03-24T05:30:19.937238+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:27:00.157605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.937742+0000","last_change":"2024-03-24T05:29:57.722165+0000","last_active":"2024-03-24T05:30:19.937742+0000","last_peered":"2024-03-24T05:30:19.937742+0000","last_clean":"2024-03-24T05:30:19.937742+0000","last_became_active":"2024-03-24T05:29:57.721846+0000","last_became_peered":"2024-03-24T05:29:57.721846+0000","last_unstale":"2024-03-24T05:30:19.937742+0000","last_undegraded":"2024-03-24T05:30:19.937742+0000","last_fullsized":"2024-03-24T05:30:19.937742+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:37:57.188558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.940071+0000","last_change":"2024-03-24T05:29:51.691612+0000","last_active":"2024-03-24T05:30:19.940071+0000","last_peered":"2024-03-24T05:30:19.940071+0000","last_clean":"2024-03-24T05:30:19.940071+0000","last_became_active":"2024-03-24T05:29:51.690596+0000","last_became_peered":"2024-03-24T05:29:51.690596+0000","last_unstale":"2024-03-24T05:30:19.940071+0000","last_undegraded":"2024-03-24T05:30:19.940071+0000","last_fullsized":"2024-03-24T05:30:19.940071+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:56.485216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938971+0000","last_change":"2024-03-24T05:29:53.699585+0000","last_active":"2024-03-24T05:30:19.938971+0000","last_peered":"2024-03-24T05:30:19.938971+0000","last_clean":"2024-03-24T05:30:19.938971+0000","last_became_active":"2024-03-24T05:29:53.699232+0000","last_became_peered":"2024-03-24T05:29:53.699232+0000","last_unstale":"2024-03-24T05:30:19.938971+0000","last_undegraded":"2024-03-24T05:30:19.938971+0000","last_fullsized":"2024-03-24T05:30:19.938971+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:36:16.813779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938434+0000","last_change":"2024-03-24T05:29:55.715215+0000","last_active":"2024-03-24T05:30:19.938434+0000","last_peered":"2024-03-24T05:30:19.938434+0000","last_clean":"2024-03-24T05:30:19.938434+0000","last_became_active":"2024-03-24T05:29:55.714875+0000","last_became_peered":"2024-03-24T05:29:55.714875+0000","last_unstale":"2024-03-24T05:30:19.938434+0000","last_undegraded":"2024-03-24T05:30:19.938434+0000","last_fullsized":"2024-03-24T05:30:19.938434+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:43.320858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.936695+0000","last_change":"2024-03-24T05:29:57.724720+0000","last_active":"2024-03-24T05:30:19.936695+0000","last_peered":"2024-03-24T05:30:19.936695+0000","last_clean":"2024-03-24T05:30:19.936695+0000","last_became_active":"2024-03-24T05:29:57.723868+0000","last_became_peered":"2024-03-24T05:29:57.723868+0000","last_unstale":"2024-03-24T05:30:19.936695+0000","last_undegraded":"2024-03-24T05:30:19.936695+0000","last_fullsized":"2024-03-24T05:30:19.936695+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:56.701913+0000","last_clean_scrub_stamp":"2024-03-24T05:29:56.701913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:01:21.433075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.942082+0000","last_change":"2024-03-24T05:29:51.692856+0000","last_active":"2024-03-24T05:30:19.942082+0000","last_peered":"2024-03-24T05:30:19.942082+0000","last_clean":"2024-03-24T05:30:19.942082+0000","last_became_active":"2024-03-24T05:29:51.692305+0000","last_became_peered":"2024-03-24T05:29:51.692305+0000","last_unstale":"2024-03-24T05:30:19.942082+0000","last_undegraded":"2024-03-24T05:30:19.942082+0000","last_fullsized":"2024-03-24T05:30:19.942082+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:50.673207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:50.673207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:27.885189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.939478+0000","last_change":"2024-03-24T05:29:53.698436+0000","last_active":"2024-03-24T05:30:19.939478+0000","last_peered":"2024-03-24T05:30:19.939478+0000","last_clean":"2024-03-24T05:30:19.939478+0000","last_became_active":"2024-03-24T05:29:53.698128+0000","last_became_peered":"2024-03-24T05:29:53.698128+0000","last_unstale":"2024-03-24T05:30:19.939478+0000","last_undegraded":"2024-03-24T05:30:19.939478+0000","last_fullsized":"2024-03-24T05:30:19.939478+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:52.678207+0000","last_clean_scrub_stamp":"2024-03-24T05:29:52.678207+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:26:45.435035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T05:30:19.938869+0000","last_change":"2024-03-24T05:29:55.716929+0000","last_active":"2024-03-24T05:30:19.938869+0000","last_peered":"2024-03-24T05:30:19.938869+0000","last_clean":"2024-03-24T05:30:19.938869+0000","last_became_active":"2024-03-24T05:29:55.716550+0000","last_became_peered":"2024-03-24T05:29:55.716550+0000","last_unstale":"2024-03-24T05:30:19.938869+0000","last_undegraded":"2024-03-24T05:30:19.938869+0000","last_fullsized":"2024-03-24T05:30:19.938869+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T05:29:54.686508+0000","last_clean_scrub_stamp":"2024-03-24T05:29:54.686508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:02.900015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":587,"ondisk_log_size":587,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411341,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738392,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T05:30:35.668 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T05:30:35.668 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T05:30:35.668 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-24T05:30:35.668 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-24T05:30:36.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:36 smithi023 ceph-mon[25869]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T05:30:36.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:36 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T05:30:36.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:36 smithi063 ceph-mon[30595]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T05:30:36.304 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:36 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T05:30:36.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:36 smithi143 ceph-mon[30288]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T05:30:36.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:36 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T05:30:36.492 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:30:37.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:37 smithi063 ceph-mon[30595]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T05:30:37.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:37 smithi143 ceph-mon[30288]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T05:30:37.320 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:37 smithi023 ceph-mon[25869]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T05:30:38.043 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:37 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: ::ffff:172.21.15.63 - - [24/Mar/2024:05:30:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T05:30:39.233 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:39 smithi023 ceph-mon[25869]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T05:30:39.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:39 smithi063 ceph-mon[30595]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T05:30:39.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:39 smithi143 ceph-mon[30288]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T05:30:40.052 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T05:30:40.052 INFO:teuthology.orchestra.run.smithi023.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T05:30:41.058 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-24T05:30:41.058 INFO:tasks.cephadm:Setup complete, yielding 2024-03-24T05:30:41.058 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T05:30:41.068 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-24T05:30:41.069 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-24T05:30:41.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:41 smithi063 ceph-mon[30595]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:41.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:41 smithi063 ceph-mon[30595]: from='client.? 172.21.15.23:0/3174555599' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T05:30:41.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:41 smithi143 ceph-mon[30288]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:41.307 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:41 smithi143 ceph-mon[30288]: from='client.? 172.21.15.23:0/3174555599' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T05:30:41.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:41 smithi023 ceph-mon[25869]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:41.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:41 smithi023 ceph-mon[25869]: from='client.? 172.21.15.23:0/3174555599' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T05:30:41.330 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 24 05:30:41 smithi023 systemd[1]: Started Ceph alertmanager.a for e2aa55c2-e99e-11ee-95cd-87774f69a715. 2024-03-24T05:30:41.331 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:30:41.583 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:41 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: [24/Mar/2024:05:30:41] ENGINE Bus STOPPING 2024-03-24T05:30:41.583 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:41 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: [24/Mar/2024:05:30:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T05:30:41.583 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:41 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: [24/Mar/2024:05:30:41] ENGINE Bus STOPPED 2024-03-24T05:30:41.583 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:41 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: [24/Mar/2024:05:30:41] ENGINE Bus STARTING 2024-03-24T05:30:41.583 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:41 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: [24/Mar/2024:05:30:41] ENGINE Serving on http://:::9283 2024-03-24T05:30:41.583 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:41 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: [24/Mar/2024:05:30:41] ENGINE Bus STARTED 2024-03-24T05:30:42.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:42 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:42.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:42 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:42.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:42 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:42.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:42 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:42.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:42 smithi023 ceph-mon[25869]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T05:30:42.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:42 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:42.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:42 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:42.330 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:42 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T05:30:42.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:42 smithi023 ceph-mon[25869]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T05:30:42.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:42 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:42.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:42 smithi023 ceph-mon[25869]: Deploying daemon grafana.a on smithi063 2024-03-24T05:30:42.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:42 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:42.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:42 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:42.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:42 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:42.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:42 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:42.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:42 smithi063 ceph-mon[30595]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T05:30:42.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:42 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:42.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:42 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:42.555 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:42 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T05:30:42.555 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:42 smithi063 ceph-mon[30595]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T05:30:42.555 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:42 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:42.555 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:42 smithi063 ceph-mon[30595]: Deploying daemon grafana.a on smithi063 2024-03-24T05:30:42.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:42 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:42.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:42 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:42.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:42 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:42.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:42 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:42.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:42 smithi143 ceph-mon[30288]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T05:30:42.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:42 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:42.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:42 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:42.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:42 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T05:30:42.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:42 smithi143 ceph-mon[30288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T05:30:42.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:42 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:42.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:42 smithi143 ceph-mon[30288]: Deploying daemon grafana.a on smithi063 2024-03-24T05:30:42.785 INFO:teuthology.orchestra.run.smithi023.stdout:167 167 2024-03-24T05:30:43.013 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 24 05:30:42 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-alertmanager-a[55086]: ts=2024-03-24T05:30:42.730Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000219172s 2024-03-24T05:30:43.267 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- bash -c 'ceph orch status' 2024-03-24T05:30:43.511 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:30:43.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:43 smithi023 ceph-mon[25869]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:43.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:43 smithi063 ceph-mon[30595]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:43.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:43 smithi143 ceph-mon[30288]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:45.362 INFO:teuthology.orchestra.run.smithi023.stdout:Backend: cephadm 2024-03-24T05:30:45.363 INFO:teuthology.orchestra.run.smithi023.stdout:Available: Yes 2024-03-24T05:30:45.363 INFO:teuthology.orchestra.run.smithi023.stdout:Paused: No 2024-03-24T05:30:45.610 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:45 smithi023 ceph-mon[25869]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:45.611 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:45 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:45.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:45 smithi063 ceph-mon[30595]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:45.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:45 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:45.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:45 smithi143 ceph-mon[30288]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:45.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:45 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:45.929 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-24T05:30:46.169 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:30:46.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:46 smithi023 ceph-mon[25869]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:30:46.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:46 smithi063 ceph-mon[30595]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:30:46.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:46 smithi143 ceph-mon[30288]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:30:47.373 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:47 smithi023 ceph-mon[25869]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:47.786 INFO:teuthology.orchestra.run.smithi023.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T05:30:47.786 INFO:teuthology.orchestra.run.smithi023.stdout:alertmanager.a smithi023 *:9093,9094 starting - - - - 2024-03-24T05:30:47.787 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.x smithi063 *:8443,8765 running (2m) 25s ago 2m 448M - 19.0.0-1667-gdb0330b1 4f325f8440e9 d03b80443110 2024-03-24T05:30:47.787 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.y smithi023 *:9283,8765 running (4m) 21s ago 4m 477M - 19.0.0-1667-gdb0330b1 4f325f8440e9 877f2bae6dcb 2024-03-24T05:30:47.787 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.z smithi143 *:8443,8765 running (2m) 26s ago 2m 447M - 19.0.0-1667-gdb0330b1 4f325f8440e9 47dd699bae4b 2024-03-24T05:30:47.787 INFO:teuthology.orchestra.run.smithi023.stdout:mon.a smithi023 running (4m) 21s ago 4m 53.1M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 8e5cbfa90e96 2024-03-24T05:30:47.787 INFO:teuthology.orchestra.run.smithi023.stdout:mon.b smithi063 running (2m) 25s ago 2m 42.3M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 055378e48274 2024-03-24T05:30:47.787 INFO:teuthology.orchestra.run.smithi023.stdout:mon.c smithi143 running (2m) 26s ago 2m 46.3M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 28c1a290e01f 2024-03-24T05:30:47.787 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0 smithi023 running (2m) 21s ago 2m 52.7M 15.6G 19.0.0-1667-gdb0330b1 4f325f8440e9 1e2642d25dd6 2024-03-24T05:30:47.787 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1 smithi063 running (103s) 25s ago 103s 55.2M 15.6G 19.0.0-1667-gdb0330b1 4f325f8440e9 2022b45651c2 2024-03-24T05:30:47.787 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2 smithi143 running (79s) 26s ago 79s 61.0M 16.6G 19.0.0-1667-gdb0330b1 4f325f8440e9 e3de23f43064 2024-03-24T05:30:47.787 INFO:teuthology.orchestra.run.smithi023.stdout:prometheus.a smithi063 *:9095 running (36s) 25s ago 36s 25.9M - 2.43.0 a07b618ecd1d d84f0a1ee3ac 2024-03-24T05:30:47.788 INFO:teuthology.orchestra.run.smithi023.stdout:rgw.foo.a smithi023 *:80 running (58s) 21s ago 57s 80.4M - 19.0.0-1667-gdb0330b1 4f325f8440e9 29ebffacb9c3 2024-03-24T05:30:47.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:47 smithi063 ceph-mon[30595]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:47.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:47 smithi143 ceph-mon[30288]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:48.079 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:47 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: ::ffff:172.21.15.63 - - [24/Mar/2024:05:30:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T05:30:48.342 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-24T05:30:48.581 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:30:49.689 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:49 smithi023 ceph-mon[25869]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:30:49.689 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:49 smithi023 ceph-mon[25869]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:49.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:49 smithi063 ceph-mon[30595]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:30:49.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:49 smithi063 ceph-mon[30595]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:49.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:49 smithi143 ceph-mon[30288]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:30:49.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:49 smithi143 ceph-mon[30288]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:50.625 INFO:teuthology.orchestra.run.smithi023.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-24T05:30:50.625 INFO:teuthology.orchestra.run.smithi023.stdout:alertmanager ?:9093,9094 0/1 - 56s smithi023=a;count:1 2024-03-24T05:30:50.625 INFO:teuthology.orchestra.run.smithi023.stdout:grafana ?:3000 0/1 - 53s smithi063=a;count:1 2024-03-24T05:30:50.625 INFO:teuthology.orchestra.run.smithi023.stdout:mgr 3/3 28s ago 2m smithi023=y;smithi063=x;smithi143=z;count:3 2024-03-24T05:30:50.625 INFO:teuthology.orchestra.run.smithi023.stdout:mon 3/3 28s ago 2m smithi023:172.21.15.23=a;smithi063:172.21.15.63=b;smithi143:172.21.15.143=c;count:3 2024-03-24T05:30:50.625 INFO:teuthology.orchestra.run.smithi023.stdout:node-exporter ?:9100 0/3 - 58s smithi023=a;smithi063=b;smithi143=c;count:3 2024-03-24T05:30:50.625 INFO:teuthology.orchestra.run.smithi023.stdout:osd 3 28s ago - 2024-03-24T05:30:50.625 INFO:teuthology.orchestra.run.smithi023.stdout:prometheus ?:9095 1/1 28s ago 61s smithi063=a;count:1 2024-03-24T05:30:50.626 INFO:teuthology.orchestra.run.smithi023.stdout:rgw.foo.a ?:80 1/1 24s ago 60s smithi023=foo.a;count:1 2024-03-24T05:30:50.626 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:50 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T05:30:50.709 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:50 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T05:30:50.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:50 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T05:30:50.927 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 24 05:30:50 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-alertmanager-a[55086]: ts=2024-03-24T05:30:50.734Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003766888s 2024-03-24T05:30:51.215 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-24T05:30:51.455 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:30:51.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:51 smithi143 ceph-mon[30288]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:51.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:51 smithi143 ceph-mon[30288]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:30:51.562 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:51 smithi023 ceph-mon[25869]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:51.562 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:51 smithi023 ceph-mon[25869]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:30:51.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:51 smithi063 ceph-mon[30595]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:51.804 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:51 smithi063 ceph-mon[30595]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:30:53.059 INFO:teuthology.orchestra.run.smithi023.stdout:HOST ADDR LABELS STATUS 2024-03-24T05:30:53.060 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023 172.21.15.23 2024-03-24T05:30:53.060 INFO:teuthology.orchestra.run.smithi023.stdout:smithi063 172.21.15.63 2024-03-24T05:30:53.060 INFO:teuthology.orchestra.run.smithi023.stdout:smithi143 172.21.15.143 2024-03-24T05:30:53.060 INFO:teuthology.orchestra.run.smithi023.stdout:3 hosts in cluster 2024-03-24T05:30:53.553 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:53 smithi063 ceph-mon[30595]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:53.607 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-24T05:30:53.631 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:53 smithi023 ceph-mon[25869]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:53.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:53 smithi143 ceph-mon[30288]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:53.848 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:30:54.553 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:54 smithi063 ceph-mon[30595]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:30:54.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:54 smithi023 ceph-mon[25869]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:30:54.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:54 smithi143 ceph-mon[30288]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:30:54.929 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:54 smithi063 podman[43611]: 2024-03-24 05:30:54.853744584 +0000 UTC m=+0.313595655 container start 82dd7fc3b52449c2c9edcb0a34943137aa3f24ccdd6d23c5766a244392aec709 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a, com.redhat.component=ubi8-container, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, release=239.1651231664, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , io.buildah.version=1.32.0, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., build-date=2022-04-29T11:48:11.055897, vcs-type=git, architecture=x86_64, description=Ceph Grafana Container, io.openshift.tags=base rhel8, version=8.5) 2024-03-24T05:30:55.180 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:54 smithi063 bash[43611]: 82dd7fc3b52449c2c9edcb0a34943137aa3f24ccdd6d23c5766a244392aec709 2024-03-24T05:30:55.180 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:54 smithi063 systemd[1]: Started Ceph grafana.a for e2aa55c2-e99e-11ee-95cd-87774f69a715. 2024-03-24T05:30:55.180 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=settings t=2024-03-24T05:30:55.029716375Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-24T05:30:55.181 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=settings t=2024-03-24T05:30:55.029903461Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-24T05:30:55.181 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=settings t=2024-03-24T05:30:55.029927644Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-24T05:30:55.181 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=settings t=2024-03-24T05:30:55.029935143Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-24T05:30:55.181 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=settings t=2024-03-24T05:30:55.029941202Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-24T05:30:55.181 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=settings t=2024-03-24T05:30:55.029947248Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-24T05:30:55.181 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=settings t=2024-03-24T05:30:55.029953556Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-24T05:30:55.181 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=settings t=2024-03-24T05:30:55.029959578Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-24T05:30:55.181 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=settings t=2024-03-24T05:30:55.029965453Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-24T05:30:55.181 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=settings t=2024-03-24T05:30:55.029971162Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-24T05:30:55.181 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=settings t=2024-03-24T05:30:55.029976534Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-24T05:30:55.181 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=settings t=2024-03-24T05:30:55.029982257Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-24T05:30:55.181 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=settings t=2024-03-24T05:30:55.029987931Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-24T05:30:55.181 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=settings t=2024-03-24T05:30:55.029993671Z level=info msg="App mode production" 2024-03-24T05:30:55.182 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=sqlstore t=2024-03-24T05:30:55.030045055Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-24T05:30:55.182 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=sqlstore t=2024-03-24T05:30:55.03006324Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-24T05:30:55.182 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.030924585Z level=info msg="Starting DB migrations" 2024-03-24T05:30:55.182 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.031005135Z level=info msg="Executing migration" id="create migration_log table" 2024-03-24T05:30:55.182 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.031516051Z level=info msg="Executing migration" id="create user table" 2024-03-24T05:30:55.183 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.031956075Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-24T05:30:55.183 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.032437043Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-24T05:30:55.183 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.032897705Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-24T05:30:55.183 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.033431363Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-24T05:30:55.183 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.033836207Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-24T05:30:55.183 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.034543991Z level=info msg="Executing migration" id="create user table v2" 2024-03-24T05:30:55.183 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.034948692Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-24T05:30:55.183 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.035356408Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-24T05:30:55.183 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.035811119Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-24T05:30:55.183 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.036231721Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-24T05:30:55.183 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.036611751Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-24T05:30:55.183 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.037020993Z level=info msg="Executing migration" id="Update user table charset" 2024-03-24T05:30:55.183 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.037333782Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-24T05:30:55.184 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.037754721Z level=info msg="Executing migration" id="Add missing user data" 2024-03-24T05:30:55.184 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.038077612Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-24T05:30:55.184 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.038538854Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-24T05:30:55.184 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.038946343Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-24T05:30:55.184 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.039368409Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-24T05:30:55.184 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.040509478Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-24T05:30:55.184 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.040942998Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-24T05:30:55.184 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.041415775Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-24T05:30:55.184 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.041852299Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-24T05:30:55.184 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.042745089Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-24T05:30:55.184 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.043609379Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-24T05:30:55.184 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.044451397Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-24T05:30:55.184 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.044875014Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-24T05:30:55.184 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.045783115Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-24T05:30:55.185 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.046642599Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-24T05:30:55.185 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.04748698Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-24T05:30:55.185 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.048316497Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-24T05:30:55.185 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.049100475Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-24T05:30:55.185 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.049945516Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-24T05:30:55.185 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.050787361Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-24T05:30:55.185 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.051220348Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-24T05:30:55.185 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.052085904Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-24T05:30:55.185 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.052951879Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-24T05:30:55.185 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.053829851Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-24T05:30:55.185 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.054713701Z level=info msg="Executing migration" id="create star table" 2024-03-24T05:30:55.185 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.055563905Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-24T05:30:55.185 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.056422458Z level=info msg="Executing migration" id="create org table v1" 2024-03-24T05:30:55.185 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.05734615Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-24T05:30:55.186 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.058225661Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-24T05:30:55.186 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.059076771Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-24T05:30:55.186 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.059950827Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-24T05:30:55.186 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.06075455Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-24T05:30:55.186 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.061214148Z level=info msg="Executing migration" id="Update org table charset" 2024-03-24T05:30:55.186 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.062016009Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-24T05:30:55.186 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.062878843Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-24T05:30:55.186 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.063780562Z level=info msg="Executing migration" id="create dashboard table" 2024-03-24T05:30:55.186 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.066487708Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-24T05:30:55.186 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.069270862Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-24T05:30:55.186 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.07220471Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-24T05:30:55.186 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.075028321Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-24T05:30:55.186 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.077819229Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-24T05:30:55.186 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.080660296Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-24T05:30:55.186 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.083498505Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-24T05:30:55.187 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.086365583Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-24T05:30:55.187 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.089188116Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-24T05:30:55.187 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.092078788Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-24T05:30:55.188 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.095098516Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-24T05:30:55.188 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.09801783Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-24T05:30:55.188 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.100922635Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-24T05:30:55.188 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.103680696Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-24T05:30:55.188 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.10665453Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-24T05:30:55.188 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.109492847Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-24T05:30:55.188 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.112381568Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-24T05:30:55.188 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.115061114Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-24T05:30:55.188 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.117806109Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-24T05:30:55.188 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.120332348Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-24T05:30:55.188 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.122945544Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-24T05:30:55.188 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.125541265Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-24T05:30:55.188 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.126075611Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-24T05:30:55.189 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.12832827Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-24T05:30:55.189 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.131112674Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-24T05:30:55.189 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.133821251Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-24T05:30:55.189 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.136437429Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-24T05:30:55.189 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.13913647Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-24T05:30:55.189 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.141773801Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-24T05:30:55.189 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.144558863Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-24T05:30:55.189 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.147215961Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-24T05:30:55.189 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.149864219Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-24T05:30:55.189 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.152507804Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-24T05:30:55.189 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.155094683Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-24T05:30:55.189 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.157714003Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-24T05:30:55.189 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.160268731Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-24T05:30:55.189 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.161012614Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-24T05:30:55.189 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.161869712Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-24T05:30:55.190 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.162688951Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-24T05:30:55.190 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.163555347Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-24T05:30:55.190 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.164319511Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-24T05:30:55.190 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.165185261Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-24T05:30:55.190 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.166040804Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-24T05:30:55.190 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.166849958Z level=info msg="Executing migration" id="create data_source table" 2024-03-24T05:30:55.190 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.1676981Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-24T05:30:55.190 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.168491629Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-24T05:30:55.190 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.168907935Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-24T05:30:55.190 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.169767702Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-24T05:30:55.190 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.170630448Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-24T05:30:55.190 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.171773763Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-24T05:30:55.190 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.172640616Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-24T05:30:55.190 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.173526862Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-24T05:30:55.190 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.17436185Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-24T05:30:55.191 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.175222807Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-24T05:30:55.191 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.176014515Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-24T05:30:55.191 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.176868303Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-24T05:30:55.191 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.177671036Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-24T05:30:55.191 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.177995993Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-24T05:30:55.191 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.178551793Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-24T05:30:55.191 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.179384891Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-24T05:30:55.191 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.179696601Z level=info msg="Executing migration" id="Add uid column" 2024-03-24T05:30:55.191 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.180203646Z level=info msg="Executing migration" id="Update uid value" 2024-03-24T05:30:55.191 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.180541999Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-24T05:30:55.191 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.181051016Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-24T05:30:55.191 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.181514733Z level=info msg="Executing migration" id="create api_key table" 2024-03-24T05:30:55.191 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.181954287Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-24T05:30:55.191 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.182814166Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-24T05:30:55.191 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.183667653Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-24T05:30:55.191 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.184524775Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-24T05:30:55.192 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.184984014Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-24T05:30:55.192 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.185842044Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-24T05:30:55.192 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.186689831Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-24T05:30:55.431 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-mon[30595]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:55.431 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:55.431 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:55.431 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:55.431 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:55.432 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.187820558Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-24T05:30:55.432 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.1886622Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-24T05:30:55.432 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.189502049Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-24T05:30:55.432 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.190387851Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-24T05:30:55.432 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.19083377Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-24T05:30:55.432 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.191655657Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-24T05:30:55.432 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.192039349Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-24T05:30:55.432 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.192514449Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-24T05:30:55.432 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.193378632Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-24T05:30:55.432 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.193931951Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-24T05:30:55.432 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.194716865Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-24T05:30:55.432 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.195277405Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-24T05:30:55.433 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.195859458Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-24T05:30:55.433 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.196686288Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-24T05:30:55.433 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.197075352Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-24T05:30:55.433 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.197508733Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-24T05:30:55.433 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.198416175Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-24T05:30:55.433 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.199250174Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-24T05:30:55.433 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.200086208Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-24T05:30:55.433 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.200886555Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-24T05:30:55.433 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.201733945Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-24T05:30:55.433 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.202627273Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-24T05:30:55.433 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.203458452Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-24T05:30:55.433 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.203791984Z level=info msg="Executing migration" id="create quota table v1" 2024-03-24T05:30:55.433 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.204676276Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-24T05:30:55.434 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.20555718Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-24T05:30:55.434 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.206395455Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-24T05:30:55.434 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.206818547Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-24T05:30:55.434 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.207701528Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-24T05:30:55.434 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.208529903Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-24T05:30:55.434 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.209379733Z level=info msg="Executing migration" id="create session table" 2024-03-24T05:30:55.434 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.209850081Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-24T05:30:55.434 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.212519575Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-24T05:30:55.434 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.212853993Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-24T05:30:55.434 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.215432729Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-24T05:30:55.434 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.218200924Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-24T05:30:55.434 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.22104592Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-24T05:30:55.435 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.223887315Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-24T05:30:55.436 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.22668561Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-24T05:30:55.436 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.229631483Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-24T05:30:55.436 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.230041982Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-24T05:30:55.436 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.232911293Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-24T05:30:55.436 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.23591847Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-24T05:30:55.436 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.238613202Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-24T05:30:55.436 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.239244213Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-24T05:30:55.436 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.242008627Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-24T05:30:55.436 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.244786703Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-24T05:30:55.437 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.24747273Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-24T05:30:55.437 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.24797154Z level=info msg="Executing migration" id="create alert table v1" 2024-03-24T05:30:55.437 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.25049185Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-24T05:30:55.437 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.253135425Z level=info msg="Executing migration" id="add index alert state" 2024-03-24T05:30:55.437 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.255613447Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-24T05:30:55.437 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.256055577Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-24T05:30:55.437 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.258701936Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-24T05:30:55.437 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.261279754Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-24T05:30:55.437 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.263955769Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-24T05:30:55.437 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.265445456Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-24T05:30:55.437 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.267981727Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-24T05:30:55.437 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.270694681Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-24T05:30:55.437 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.273170806Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-24T05:30:55.437 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.273574852Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-24T05:30:55.437 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.27624707Z level=info msg="Executing migration" id="Add column is_default" 2024-03-24T05:30:55.438 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.276909314Z level=info msg="Executing migration" id="Add column frequency" 2024-03-24T05:30:55.438 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.278891814Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-24T05:30:55.438 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.281429433Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-24T05:30:55.438 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.282095345Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-24T05:30:55.438 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.282937994Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-24T05:30:55.438 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.283698173Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-24T05:30:55.438 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.284031734Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-24T05:30:55.438 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.284564442Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-24T05:30:55.438 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.285028268Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-24T05:30:55.438 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.285884883Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-24T05:30:55.438 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.28672202Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-24T05:30:55.438 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.287566023Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-24T05:30:55.438 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.288244462Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-24T05:30:55.438 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.289070976Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-24T05:30:55.438 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.289905735Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-24T05:30:55.438 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.290719343Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-24T05:30:55.439 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.291155984Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-24T05:30:55.439 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.291989757Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-24T05:30:55.439 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.292829022Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-24T05:30:55.439 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.293680843Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-24T05:30:55.439 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.29414804Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-24T05:30:55.439 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.294994326Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-24T05:30:55.439 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.295824999Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-24T05:30:55.439 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.29667384Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-24T05:30:55.439 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.297135326Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-24T05:30:55.439 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.297963127Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-24T05:30:55.439 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.298805002Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-24T05:30:55.439 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.299689058Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-24T05:30:55.439 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.300488793Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-24T05:30:55.439 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.301373857Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-24T05:30:55.439 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.302270396Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-24T05:30:55.439 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.303105218Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-24T05:30:55.440 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.303529449Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-24T05:30:55.440 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.304395969Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-24T05:30:55.440 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.305236665Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-24T05:30:55.440 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.306827688Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-24T05:30:55.440 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.307241147Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-24T05:30:55.440 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.308089859Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-24T05:30:55.440 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.308964718Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-24T05:30:55.440 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.311646534Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-24T05:30:55.440 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.311992502Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-24T05:30:55.440 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.314807144Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-24T05:30:55.440 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.31768337Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-24T05:30:55.440 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.320453229Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-24T05:30:55.440 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.32091287Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-24T05:30:55.440 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.323612231Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-24T05:30:55.440 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.324322933Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-24T05:30:55.441 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.326475424Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-24T05:30:55.441 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.329399075Z level=info msg="Executing migration" id="Move region to single row" 2024-03-24T05:30:55.441 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.329794282Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-24T05:30:55.441 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.33259191Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-24T05:30:55.441 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.333049136Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-24T05:30:55.441 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.335322266Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-24T05:30:55.441 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.338072128Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-24T05:30:55.441 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.338499579Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-24T05:30:55.441 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.341337018Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-24T05:30:55.441 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.344132853Z level=info msg="Executing migration" id="create test_data table" 2024-03-24T05:30:55.441 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.346821468Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-24T05:30:55.441 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.347228498Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-24T05:30:55.441 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.349935162Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-24T05:30:55.441 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.35245955Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-24T05:30:55.441 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.355113284Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-24T05:30:55.441 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.355465734Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-24T05:30:55.442 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.35811403Z level=info msg="Executing migration" id="create team table" 2024-03-24T05:30:55.442 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.358525004Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-24T05:30:55.442 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.360803688Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-24T05:30:55.442 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.363299196Z level=info msg="Executing migration" id="create team member table" 2024-03-24T05:30:55.442 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.363706323Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-24T05:30:55.442 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.366401887Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-24T05:30:55.442 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.369142839Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-24T05:30:55.442 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.37171377Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-24T05:30:55.442 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.372487079Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-24T05:30:55.442 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.373332422Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-24T05:30:55.442 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.374230527Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-24T05:30:55.442 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.375273078Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-24T05:30:55.442 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.375728842Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-24T05:30:55.442 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.376585838Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-24T05:30:55.442 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.377040225Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-24T05:30:55.443 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.377927142Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-24T05:30:55.443 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.378380923Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-24T05:30:55.443 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.378823339Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-24T05:30:55.443 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.379266562Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-24T05:30:55.443 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.379685117Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-24T05:30:55.443 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.379977768Z level=info msg="Executing migration" id="create tag table" 2024-03-24T05:30:55.443 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.38039276Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-24T05:30:55.443 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.380840738Z level=info msg="Executing migration" id="create login attempt table" 2024-03-24T05:30:55.443 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.381245296Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-24T05:30:55.443 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.381687781Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-24T05:30:55.443 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.382090529Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-24T05:30:55.443 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.384062947Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-24T05:30:55.443 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.384483654Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-24T05:30:55.443 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.384948752Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-24T05:30:55.444 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.38534033Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-24T05:30:55.444 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.385715221Z level=info msg="Executing migration" id="create user auth table" 2024-03-24T05:30:55.444 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.386076513Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-24T05:30:55.444 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.38652164Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-24T05:30:55.444 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.386831862Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-24T05:30:55.444 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.387604068Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-24T05:30:55.444 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.388398419Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-24T05:30:55.444 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.389168373Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-24T05:30:55.444 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.389945587Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-24T05:30:55.444 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.390398949Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-24T05:30:55.444 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.391168607Z level=info msg="Executing migration" id="create server_lock table" 2024-03-24T05:30:55.444 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.391589022Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-24T05:30:55.444 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.392036505Z level=info msg="Executing migration" id="create user auth token table" 2024-03-24T05:30:55.444 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.392540405Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-24T05:30:55.444 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.393323577Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-24T05:30:55.445 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.393767554Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-24T05:30:55.445 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.394190982Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-24T05:30:55.445 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.394949491Z level=info msg="Executing migration" id="create cache_data table" 2024-03-24T05:30:55.445 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.395369259Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-24T05:30:55.445 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.395796747Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-24T05:30:55.445 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.39621164Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-24T05:30:55.445 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.397011844Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-24T05:30:55.445 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.39731578Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-24T05:30:55.445 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.397619986Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-24T05:30:55.445 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.398049492Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-24T05:30:55.445 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.400845453Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-24T05:30:55.445 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.403659376Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-24T05:30:55.445 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.40645681Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-24T05:30:55.445 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.406916397Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-24T05:30:55.445 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.409757077Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-24T05:30:55.446 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.412451059Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-24T05:30:55.446 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.415295128Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-24T05:30:55.446 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.418205453Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-24T05:30:55.446 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.421096116Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-24T05:30:55.446 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.423932367Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-24T05:30:55.446 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.426833541Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-24T05:30:55.446 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.429824025Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-24T05:30:55.446 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.432723122Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-24T05:30:55.446 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.435506494Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-24T05:30:55.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:55 smithi023 ceph-mon[25869]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:55.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:55 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:55.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:55 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:55.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:55 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:55.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:55 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:55.683 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.438390314Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-24T05:30:55.684 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.441166691Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-24T05:30:55.684 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.443946922Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-24T05:30:55.684 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.446925597Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-24T05:30:55.684 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.449505848Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-24T05:30:55.684 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.452217124Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-24T05:30:55.684 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.454898752Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-24T05:30:55.684 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.458453464Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-24T05:30:55.684 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.461927458Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-24T05:30:55.684 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.464845553Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-24T05:30:55.684 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.467489795Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-24T05:30:55.684 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.470290488Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-24T05:30:55.684 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.470794058Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-24T05:30:55.684 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.473526211Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-24T05:30:55.685 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.476265731Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-24T05:30:55.685 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.478798972Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-24T05:30:55.685 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.48161307Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-24T05:30:55.685 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.484224037Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-24T05:30:55.686 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.487050322Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-24T05:30:55.686 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.489434903Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-24T05:30:55.686 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.492248536Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-24T05:30:55.686 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.494701956Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-24T05:30:55.686 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.497413476Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-24T05:30:55.686 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.498244584Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-24T05:30:55.686 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.499106123Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-24T05:30:55.686 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.499952294Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-24T05:30:55.686 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.500800198Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-24T05:30:55.686 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.501597261Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-24T05:30:55.686 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.502577067Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-24T05:30:55.686 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.503271859Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-24T05:30:55.687 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.504060093Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-24T05:30:55.687 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.50439006Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-24T05:30:55.687 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.505262056Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-24T05:30:55.687 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.506120008Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-24T05:30:55.687 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.506981757Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-24T05:30:55.687 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.507851164Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-24T05:30:55.687 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.508718679Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-24T05:30:55.687 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.509627562Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-24T05:30:55.687 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.510450322Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-24T05:30:55.687 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.511312466Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-24T05:30:55.687 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.51229374Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-24T05:30:55.687 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.51339645Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-24T05:30:55.687 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.514252659Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-24T05:30:55.688 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.515113148Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-24T05:30:55.688 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.515998742Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-24T05:30:55.688 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.516876109Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-24T05:30:55.688 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.517765171Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-24T05:30:55.688 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.518632617Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-24T05:30:55.688 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.519493724Z level=info msg="Executing migration" id="create alert_image table" 2024-03-24T05:30:55.688 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.520360818Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-24T05:30:55.688 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.52119655Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-24T05:30:55.688 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.522011403Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-24T05:30:55.688 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.523013998Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-24T05:30:55.688 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.523617453Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-24T05:30:55.688 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.523698012Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-24T05:30:55.688 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.524467045Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-24T05:30:55.688 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.525351983Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-24T05:30:55.688 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.526227394Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-24T05:30:55.688 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.52634816Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-24T05:30:55.689 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.526460323Z level=info msg="alerts found to migrate" alerts=0 2024-03-24T05:30:55.689 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.527049536Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-24T05:30:55.689 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.527937086Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-24T05:30:55.689 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.528910836Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-24T05:30:55.689 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.531498885Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-24T05:30:55.689 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.534418903Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-24T05:30:55.689 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.534948778Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-24T05:30:55.689 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.537752733Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-24T05:30:55.689 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.538137877Z level=info msg="Executing migration" id="create data_keys table" 2024-03-24T05:30:55.689 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.540902593Z level=info msg="Executing migration" id="create secrets table" 2024-03-24T05:30:55.689 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.543611836Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-24T05:30:55.689 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.548298217Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-24T05:30:55.689 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.550978706Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-24T05:30:55.689 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.553498986Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-24T05:30:55.689 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.557827971Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-24T05:30:55.689 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.562212695Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-24T05:30:55.690 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.565031417Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-24T05:30:55.690 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.567674043Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-24T05:30:55.690 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.570263839Z level=info msg="Executing migration" id="create permission table" 2024-03-24T05:30:55.690 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.572978316Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-24T05:30:55.690 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.575705922Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-24T05:30:55.690 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.57812677Z level=info msg="Executing migration" id="create role table" 2024-03-24T05:30:55.690 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.580888719Z level=info msg="Executing migration" id="add column display_name" 2024-03-24T05:30:55.690 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.583367827Z level=info msg="Executing migration" id="add column group_name" 2024-03-24T05:30:55.690 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.585948402Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-24T05:30:55.690 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.588706623Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-24T05:30:55.690 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.591239581Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-24T05:30:55.690 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.592214734Z level=info msg="Executing migration" id="create team role table" 2024-03-24T05:30:55.690 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.592858939Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-24T05:30:55.690 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.593829681Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-24T05:30:55.690 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.594555486Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-24T05:30:55.691 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.595405477Z level=info msg="Executing migration" id="create user role table" 2024-03-24T05:30:55.691 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.596265657Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-24T05:30:55.691 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.597286833Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-24T05:30:55.691 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.59809446Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-24T05:30:55.691 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.598870214Z level=info msg="Executing migration" id="create builtin role table" 2024-03-24T05:30:55.691 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.599847074Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-24T05:30:55.691 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.600577491Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-24T05:30:55.691 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.601444436Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-24T05:30:55.692 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.602413717Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-24T05:30:55.692 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.603285354Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-24T05:30:55.692 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.604260638Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-24T05:30:55.692 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.605088094Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-24T05:30:55.692 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.605723428Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-24T05:30:55.692 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.606747289Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-24T05:30:55.692 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.607439301Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-24T05:30:55.692 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.608406107Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-24T05:30:55.692 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.609299005Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-24T05:30:55.692 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.610169814Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-24T05:30:55.692 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.611089814Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-24T05:30:55.692 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.611836663Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-24T05:30:55.692 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.61266755Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-24T05:30:55.692 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.613049287Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-24T05:30:55.692 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.613572175Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-24T05:30:55.693 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.614423666Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-24T05:30:55.693 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.615432655Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-24T05:30:55.693 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.616160052Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-24T05:30:55.693 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.617127896Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-24T05:30:55.693 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.618148407Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-24T05:30:55.693 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.618867894Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-24T05:30:55.693 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.619745067Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-24T05:30:55.693 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.620598319Z level=info msg="Executing migration" id="add correlation config column" 2024-03-24T05:30:55.693 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.623304309Z level=info msg="Executing migration" id="create entity_events table" 2024-03-24T05:30:55.693 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.626216965Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-24T05:30:55.693 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.629061Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-24T05:30:55.693 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.629236037Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-24T05:30:55.693 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.631770673Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-24T05:30:55.693 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.631981999Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-24T05:30:55.693 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.634492435Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-24T05:30:55.693 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.637266226Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-24T05:30:55.694 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.640126741Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-24T05:30:55.694 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.642959153Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-24T05:30:55.694 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.64594677Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-24T05:30:55.694 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.64861743Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-24T05:30:55.694 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.651621565Z level=info msg="Executing migration" id="Drop public config table" 2024-03-24T05:30:55.694 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.654551169Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-24T05:30:55.694 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.657367948Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-24T05:30:55.694 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.660061452Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-24T05:30:55.694 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.662901512Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-24T05:30:55.694 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.665635334Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-24T05:30:55.694 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.668873749Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-24T05:30:55.694 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.671771129Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-24T05:30:55.694 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.674308593Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-24T05:30:55.695 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.677084196Z level=info msg="Executing migration" id="add share column" 2024-03-24T05:30:55.695 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.679905306Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-24T05:30:55.695 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.682456808Z level=info msg="Executing migration" id="create file table" 2024-03-24T05:30:55.695 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.685117624Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-24T05:30:55.695 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.687720244Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-24T05:30:55.695 INFO:teuthology.orchestra.run.smithi023.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-24T05:30:55.695 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736103B14P0KGN 3726G No 2m ago Has a FileSystem, LVM detected 2024-03-24T05:30:55.695 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023 /dev/nvme1n1 ssd Linux_8549904116af059a0488 89.4G Yes 2m ago 2024-03-24T05:30:55.695 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023 /dev/nvme2n1 ssd Linux_28689d8b43718627a57c 89.4G Yes 2m ago 2024-03-24T05:30:55.695 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023 /dev/nvme3n1 ssd Linux_b7d9cdebba36738fb371 89.4G Yes 2m ago 2024-03-24T05:30:55.695 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023 /dev/nvme4n1 ssd Linux_cc13a3e672a650c05812 89.4G No 2m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-24T05:30:55.696 INFO:teuthology.orchestra.run.smithi023.stdout:smithi063 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5344005B400BGN 372G No 103s ago Has a FileSystem, LVM detected 2024-03-24T05:30:55.696 INFO:teuthology.orchestra.run.smithi023.stdout:smithi063 /dev/nvme1n1 ssd Linux_1162f4684f88f110d8c7 89.4G Yes 103s ago 2024-03-24T05:30:55.696 INFO:teuthology.orchestra.run.smithi023.stdout:smithi063 /dev/nvme2n1 ssd Linux_ff402ff699d2c9ece9a6 89.4G Yes 103s ago 2024-03-24T05:30:55.696 INFO:teuthology.orchestra.run.smithi023.stdout:smithi063 /dev/nvme3n1 ssd Linux_188d14ce55301b58156c 89.4G Yes 103s ago 2024-03-24T05:30:55.696 INFO:teuthology.orchestra.run.smithi023.stdout:smithi063 /dev/nvme4n1 ssd Linux_144efd7c913394e7e3ec 89.4G No 103s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-24T05:30:55.696 INFO:teuthology.orchestra.run.smithi023.stdout:smithi143 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT62330073400BGN 372G No 79s ago Has a FileSystem, LVM detected 2024-03-24T05:30:55.696 INFO:teuthology.orchestra.run.smithi023.stdout:smithi143 /dev/nvme1n1 ssd Linux_948f005413c860afb7e4 89.4G Yes 79s ago 2024-03-24T05:30:55.696 INFO:teuthology.orchestra.run.smithi023.stdout:smithi143 /dev/nvme2n1 ssd Linux_55da23698e9b0746f0c8 89.4G Yes 79s ago 2024-03-24T05:30:55.696 INFO:teuthology.orchestra.run.smithi023.stdout:smithi143 /dev/nvme3n1 ssd Linux_1f4e4e16c9dfacc88dba 89.4G Yes 79s ago 2024-03-24T05:30:55.696 INFO:teuthology.orchestra.run.smithi023.stdout:smithi143 /dev/nvme4n1 ssd Linux_8af335e728b9053b18b9 89.4G No 79s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-24T05:30:55.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:55 smithi143 ceph-mon[30288]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:55.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:55 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:55.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:55 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:55.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:55 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:55.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:55 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:30:56.054 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.690359268Z level=info msg="Executing migration" id="create file_meta table" 2024-03-24T05:30:56.054 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.693022529Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-24T05:30:56.055 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.695553151Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-24T05:30:56.055 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.698136631Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-24T05:30:56.055 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.700891556Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-24T05:30:56.055 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.703466917Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-24T05:30:56.055 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.706321472Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-24T05:30:56.055 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.708891092Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-24T05:30:56.055 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.711531453Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-24T05:30:56.055 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.714104595Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-24T05:30:56.055 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.716900234Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-24T05:30:56.056 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.717604255Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-24T05:30:56.056 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.718465813Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-24T05:30:56.056 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.719514971Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-24T05:30:56.056 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.72056031Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-24T05:30:56.056 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.721399686Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-24T05:30:56.056 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.731768929Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-24T05:30:56.056 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.732626545Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-24T05:30:56.056 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.733638882Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-24T05:30:56.057 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.737289403Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-24T05:30:56.057 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.738114002Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-24T05:30:56.057 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.739091956Z level=info msg="Executing migration" id="create folder table" 2024-03-24T05:30:56.057 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.739840278Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-24T05:30:56.057 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.740610591Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-24T05:30:56.057 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.741465545Z level=info msg="Executing migration" id="Update folder title length" 2024-03-24T05:30:56.057 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.742297361Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-24T05:30:56.057 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=migrator t=2024-03-24T05:30:55.743129707Z level=info msg="migrations completed" performed=485 skipped=0 duration=712.148721ms 2024-03-24T05:30:56.058 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=sqlstore t=2024-03-24T05:30:55.743549059Z level=info msg="Created default organization" 2024-03-24T05:30:56.058 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=plugin.loader t=2024-03-24T05:30:55.802499545Z level=info msg="Plugin registered" pluginID=input 2024-03-24T05:30:56.058 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=plugin.loader t=2024-03-24T05:30:55.832206943Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-24T05:30:56.058 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=plugin.loader t=2024-03-24T05:30:55.832288683Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-24T05:30:56.058 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=secrets t=2024-03-24T05:30:55.832385758Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-24T05:30:56.058 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=query_data t=2024-03-24T05:30:55.835344951Z level=info msg="Query Service initialization" 2024-03-24T05:30:56.058 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:55 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=live.push_http t=2024-03-24T05:30:55.838128546Z level=info msg="Live Push Gateway initialization" 2024-03-24T05:30:56.391 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-24T05:30:56.679 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:30:56.800 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:56 smithi023 ceph-mon[25869]: Deploying daemon node-exporter.a on smithi023 2024-03-24T05:30:56.800 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:56 smithi023 ceph-mon[25869]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:30:56.800 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:56 smithi023 bash[56630]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-24T05:30:56.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:56 smithi063 ceph-mon[30595]: Deploying daemon node-exporter.a on smithi023 2024-03-24T05:30:56.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:56 smithi063 ceph-mon[30595]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:30:56.804 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:56 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=infra.usagestats.collector t=2024-03-24T05:30:56.482776971Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-24T05:30:56.804 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:56 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=provisioning.datasources t=2024-03-24T05:30:56.483805495Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-24T05:30:56.804 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:56 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=provisioning.datasources t=2024-03-24T05:30:56.494967094Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-24T05:30:56.804 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:56 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=provisioning.alerting t=2024-03-24T05:30:56.506085714Z level=info msg="starting to provision alerting" 2024-03-24T05:30:56.804 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:56 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=provisioning.alerting t=2024-03-24T05:30:56.506110343Z level=info msg="finished to provision alerting" 2024-03-24T05:30:56.804 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:56 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=grafanaStorageLogger t=2024-03-24T05:30:56.506244894Z level=info msg="storage starting" 2024-03-24T05:30:56.804 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:56 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=ngalert.state.manager t=2024-03-24T05:30:56.506301747Z level=info msg="Warming state cache for startup" 2024-03-24T05:30:56.804 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:56 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=ngalert.state.manager t=2024-03-24T05:30:56.506440359Z level=info msg="State cache has been initialized" states=0 duration=137.772µs 2024-03-24T05:30:56.804 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:56 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=ticker t=2024-03-24T05:30:56.506507871Z level=info msg=starting first_tick=2024-03-24T05:31:00Z 2024-03-24T05:30:56.804 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:56 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=ngalert.multiorg.alertmanager t=2024-03-24T05:30:56.506531172Z level=info msg="starting MultiOrg Alertmanager" 2024-03-24T05:30:56.804 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:56 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=http.server t=2024-03-24T05:30:56.508187053Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-24T05:30:56.804 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:30:56 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a[43633]: logger=sqlstore.transactions t=2024-03-24T05:30:56.526127761Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-24T05:30:56.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:56 smithi143 ceph-mon[30288]: Deploying daemon node-exporter.a on smithi023 2024-03-24T05:30:56.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:56 smithi143 ceph-mon[30288]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T05:30:57.079 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:56 smithi023 bash[56630]: Getting image source signatures 2024-03-24T05:30:57.079 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:56 smithi023 bash[56630]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-24T05:30:57.079 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:56 smithi023 bash[56630]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-24T05:30:57.080 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:56 smithi023 bash[56630]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-24T05:30:57.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:57 smithi023 ceph-mon[25869]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:57.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:57 smithi063 ceph-mon[30595]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:57.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:57 smithi143 ceph-mon[30288]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:58.079 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:30:57 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: ::ffff:172.21.15.63 - - [24/Mar/2024:05:30:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T05:30:58.541 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:58 smithi023 bash[56630]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-24T05:30:58.829 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:58 smithi023 bash[56630]: Writing manifest to image destination 2024-03-24T05:30:59.259 INFO:teuthology.orchestra.run.smithi023.stdout:service_type: alertmanager 2024-03-24T05:30:59.259 INFO:teuthology.orchestra.run.smithi023.stdout:service_name: alertmanager 2024-03-24T05:30:59.259 INFO:teuthology.orchestra.run.smithi023.stdout:placement: 2024-03-24T05:30:59.259 INFO:teuthology.orchestra.run.smithi023.stdout: count: 1 2024-03-24T05:30:59.259 INFO:teuthology.orchestra.run.smithi023.stdout: hosts: 2024-03-24T05:30:59.259 INFO:teuthology.orchestra.run.smithi023.stdout: - smithi023=a 2024-03-24T05:30:59.259 INFO:teuthology.orchestra.run.smithi023.stdout:status: 2024-03-24T05:30:59.259 INFO:teuthology.orchestra.run.smithi023.stdout: created: '2024-03-24T05:29:54.621098Z' 2024-03-24T05:30:59.259 INFO:teuthology.orchestra.run.smithi023.stdout: ports: 2024-03-24T05:30:59.259 INFO:teuthology.orchestra.run.smithi023.stdout: - 9093 2024-03-24T05:30:59.259 INFO:teuthology.orchestra.run.smithi023.stdout: - 9094 2024-03-24T05:30:59.260 INFO:teuthology.orchestra.run.smithi023.stdout: running: 0 2024-03-24T05:30:59.260 INFO:teuthology.orchestra.run.smithi023.stdout: size: 1 2024-03-24T05:30:59.260 INFO:teuthology.orchestra.run.smithi023.stdout:events: 2024-03-24T05:30:59.260 INFO:teuthology.orchestra.run.smithi023.stdout:- 2024-03-24T05:30:41.262230Z service:alertmanager [INFO] "service was created" 2024-03-24T05:30:59.260 INFO:teuthology.orchestra.run.smithi023.stdout:--- 2024-03-24T05:30:59.260 INFO:teuthology.orchestra.run.smithi023.stdout:service_type: grafana 2024-03-24T05:30:59.260 INFO:teuthology.orchestra.run.smithi023.stdout:service_name: grafana 2024-03-24T05:30:59.260 INFO:teuthology.orchestra.run.smithi023.stdout:placement: 2024-03-24T05:30:59.260 INFO:teuthology.orchestra.run.smithi023.stdout: count: 1 2024-03-24T05:30:59.260 INFO:teuthology.orchestra.run.smithi023.stdout: hosts: 2024-03-24T05:30:59.260 INFO:teuthology.orchestra.run.smithi023.stdout: - smithi063=a 2024-03-24T05:30:59.260 INFO:teuthology.orchestra.run.smithi023.stdout:spec: 2024-03-24T05:30:59.260 INFO:teuthology.orchestra.run.smithi023.stdout: anonymous_access: true 2024-03-24T05:30:59.260 INFO:teuthology.orchestra.run.smithi023.stdout: protocol: https 2024-03-24T05:30:59.261 INFO:teuthology.orchestra.run.smithi023.stdout:status: 2024-03-24T05:30:59.261 INFO:teuthology.orchestra.run.smithi023.stdout: created: '2024-03-24T05:29:57.277138Z' 2024-03-24T05:30:59.261 INFO:teuthology.orchestra.run.smithi023.stdout: ports: 2024-03-24T05:30:59.261 INFO:teuthology.orchestra.run.smithi023.stdout: - 3000 2024-03-24T05:30:59.261 INFO:teuthology.orchestra.run.smithi023.stdout: running: 0 2024-03-24T05:30:59.261 INFO:teuthology.orchestra.run.smithi023.stdout: size: 1 2024-03-24T05:30:59.261 INFO:teuthology.orchestra.run.smithi023.stdout:events: 2024-03-24T05:30:59.261 INFO:teuthology.orchestra.run.smithi023.stdout:- 2024-03-24T05:30:55.032513Z service:grafana [INFO] "service was created" 2024-03-24T05:30:59.261 INFO:teuthology.orchestra.run.smithi023.stdout:--- 2024-03-24T05:30:59.261 INFO:teuthology.orchestra.run.smithi023.stdout:service_type: mgr 2024-03-24T05:30:59.261 INFO:teuthology.orchestra.run.smithi023.stdout:service_name: mgr 2024-03-24T05:30:59.261 INFO:teuthology.orchestra.run.smithi023.stdout:placement: 2024-03-24T05:30:59.261 INFO:teuthology.orchestra.run.smithi023.stdout: count: 3 2024-03-24T05:30:59.261 INFO:teuthology.orchestra.run.smithi023.stdout: hosts: 2024-03-24T05:30:59.262 INFO:teuthology.orchestra.run.smithi023.stdout: - smithi023=y 2024-03-24T05:30:59.262 INFO:teuthology.orchestra.run.smithi023.stdout: - smithi063=x 2024-03-24T05:30:59.262 INFO:teuthology.orchestra.run.smithi023.stdout: - smithi143=z 2024-03-24T05:30:59.262 INFO:teuthology.orchestra.run.smithi023.stdout:status: 2024-03-24T05:30:59.262 INFO:teuthology.orchestra.run.smithi023.stdout: created: '2024-03-24T05:28:16.419809Z' 2024-03-24T05:30:59.262 INFO:teuthology.orchestra.run.smithi023.stdout: last_refresh: '2024-03-24T05:30:21.751848Z' 2024-03-24T05:30:59.262 INFO:teuthology.orchestra.run.smithi023.stdout: running: 3 2024-03-24T05:30:59.262 INFO:teuthology.orchestra.run.smithi023.stdout: size: 3 2024-03-24T05:30:59.262 INFO:teuthology.orchestra.run.smithi023.stdout:--- 2024-03-24T05:30:59.262 INFO:teuthology.orchestra.run.smithi023.stdout:service_type: mon 2024-03-24T05:30:59.262 INFO:teuthology.orchestra.run.smithi023.stdout:service_name: mon 2024-03-24T05:30:59.262 INFO:teuthology.orchestra.run.smithi023.stdout:placement: 2024-03-24T05:30:59.262 INFO:teuthology.orchestra.run.smithi023.stdout: count: 3 2024-03-24T05:30:59.263 INFO:teuthology.orchestra.run.smithi023.stdout: hosts: 2024-03-24T05:30:59.263 INFO:teuthology.orchestra.run.smithi023.stdout: - smithi023:172.21.15.23=a 2024-03-24T05:30:59.263 INFO:teuthology.orchestra.run.smithi023.stdout: - smithi063:172.21.15.63=b 2024-03-24T05:30:59.263 INFO:teuthology.orchestra.run.smithi023.stdout: - smithi143:172.21.15.143=c 2024-03-24T05:30:59.263 INFO:teuthology.orchestra.run.smithi023.stdout:status: 2024-03-24T05:30:59.263 INFO:teuthology.orchestra.run.smithi023.stdout: created: '2024-03-24T05:27:50.989006Z' 2024-03-24T05:30:59.263 INFO:teuthology.orchestra.run.smithi023.stdout: last_refresh: '2024-03-24T05:30:21.751738Z' 2024-03-24T05:30:59.263 INFO:teuthology.orchestra.run.smithi023.stdout: running: 3 2024-03-24T05:30:59.263 INFO:teuthology.orchestra.run.smithi023.stdout: size: 3 2024-03-24T05:30:59.263 INFO:teuthology.orchestra.run.smithi023.stdout:--- 2024-03-24T05:30:59.263 INFO:teuthology.orchestra.run.smithi023.stdout:service_type: node-exporter 2024-03-24T05:30:59.263 INFO:teuthology.orchestra.run.smithi023.stdout:service_name: node-exporter 2024-03-24T05:30:59.263 INFO:teuthology.orchestra.run.smithi023.stdout:placement: 2024-03-24T05:30:59.263 INFO:teuthology.orchestra.run.smithi023.stdout: count: 3 2024-03-24T05:30:59.264 INFO:teuthology.orchestra.run.smithi023.stdout: hosts: 2024-03-24T05:30:59.264 INFO:teuthology.orchestra.run.smithi023.stdout: - smithi023=a 2024-03-24T05:30:59.264 INFO:teuthology.orchestra.run.smithi023.stdout: - smithi063=b 2024-03-24T05:30:59.264 INFO:teuthology.orchestra.run.smithi023.stdout: - smithi143=c 2024-03-24T05:30:59.264 INFO:teuthology.orchestra.run.smithi023.stdout:status: 2024-03-24T05:30:59.264 INFO:teuthology.orchestra.run.smithi023.stdout: created: '2024-03-24T05:29:51.951531Z' 2024-03-24T05:30:59.264 INFO:teuthology.orchestra.run.smithi023.stdout: ports: 2024-03-24T05:30:59.264 INFO:teuthology.orchestra.run.smithi023.stdout: - 9100 2024-03-24T05:30:59.264 INFO:teuthology.orchestra.run.smithi023.stdout: running: 0 2024-03-24T05:30:59.264 INFO:teuthology.orchestra.run.smithi023.stdout: size: 3 2024-03-24T05:30:59.264 INFO:teuthology.orchestra.run.smithi023.stdout:--- 2024-03-24T05:30:59.264 INFO:teuthology.orchestra.run.smithi023.stdout:service_type: osd 2024-03-24T05:30:59.264 INFO:teuthology.orchestra.run.smithi023.stdout:service_name: osd 2024-03-24T05:30:59.264 INFO:teuthology.orchestra.run.smithi023.stdout:unmanaged: true 2024-03-24T05:30:59.265 INFO:teuthology.orchestra.run.smithi023.stdout:spec: 2024-03-24T05:30:59.265 INFO:teuthology.orchestra.run.smithi023.stdout: filter_logic: AND 2024-03-24T05:30:59.265 INFO:teuthology.orchestra.run.smithi023.stdout: objectstore: bluestore 2024-03-24T05:30:59.265 INFO:teuthology.orchestra.run.smithi023.stdout:status: 2024-03-24T05:30:59.265 INFO:teuthology.orchestra.run.smithi023.stdout: container_image_id: 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d 2024-03-24T05:30:59.265 INFO:teuthology.orchestra.run.smithi023.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T05:30:59.265 INFO:teuthology.orchestra.run.smithi023.stdout: last_refresh: '2024-03-24T05:30:21.751930Z' 2024-03-24T05:30:59.265 INFO:teuthology.orchestra.run.smithi023.stdout: running: 3 2024-03-24T05:30:59.265 INFO:teuthology.orchestra.run.smithi023.stdout: size: 3 2024-03-24T05:30:59.265 INFO:teuthology.orchestra.run.smithi023.stdout:--- 2024-03-24T05:30:59.265 INFO:teuthology.orchestra.run.smithi023.stdout:service_type: prometheus 2024-03-24T05:30:59.265 INFO:teuthology.orchestra.run.smithi023.stdout:service_name: prometheus 2024-03-24T05:30:59.265 INFO:teuthology.orchestra.run.smithi023.stdout:placement: 2024-03-24T05:30:59.265 INFO:teuthology.orchestra.run.smithi023.stdout: count: 1 2024-03-24T05:30:59.266 INFO:teuthology.orchestra.run.smithi023.stdout: hosts: 2024-03-24T05:30:59.266 INFO:teuthology.orchestra.run.smithi023.stdout: - smithi063=a 2024-03-24T05:30:59.266 INFO:teuthology.orchestra.run.smithi023.stdout:status: 2024-03-24T05:30:59.266 INFO:teuthology.orchestra.run.smithi023.stdout: created: '2024-03-24T05:29:49.216809Z' 2024-03-24T05:30:59.266 INFO:teuthology.orchestra.run.smithi023.stdout: last_refresh: '2024-03-24T05:30:22.384854Z' 2024-03-24T05:30:59.266 INFO:teuthology.orchestra.run.smithi023.stdout: ports: 2024-03-24T05:30:59.266 INFO:teuthology.orchestra.run.smithi023.stdout: - 9095 2024-03-24T05:30:59.266 INFO:teuthology.orchestra.run.smithi023.stdout: running: 1 2024-03-24T05:30:59.266 INFO:teuthology.orchestra.run.smithi023.stdout: size: 1 2024-03-24T05:30:59.266 INFO:teuthology.orchestra.run.smithi023.stdout:--- 2024-03-24T05:30:59.266 INFO:teuthology.orchestra.run.smithi023.stdout:service_type: rgw 2024-03-24T05:30:59.266 INFO:teuthology.orchestra.run.smithi023.stdout:service_id: foo.a 2024-03-24T05:30:59.266 INFO:teuthology.orchestra.run.smithi023.stdout:service_name: rgw.foo.a 2024-03-24T05:30:59.267 INFO:teuthology.orchestra.run.smithi023.stdout:placement: 2024-03-24T05:30:59.267 INFO:teuthology.orchestra.run.smithi023.stdout: count: 1 2024-03-24T05:30:59.267 INFO:teuthology.orchestra.run.smithi023.stdout: hosts: 2024-03-24T05:30:59.267 INFO:teuthology.orchestra.run.smithi023.stdout: - smithi023=foo.a 2024-03-24T05:30:59.267 INFO:teuthology.orchestra.run.smithi023.stdout:status: 2024-03-24T05:30:59.267 INFO:teuthology.orchestra.run.smithi023.stdout: created: '2024-03-24T05:29:50.035279Z' 2024-03-24T05:30:59.267 INFO:teuthology.orchestra.run.smithi023.stdout: last_refresh: '2024-03-24T05:30:26.461599Z' 2024-03-24T05:30:59.267 INFO:teuthology.orchestra.run.smithi023.stdout: ports: 2024-03-24T05:30:59.267 INFO:teuthology.orchestra.run.smithi023.stdout: - 80 2024-03-24T05:30:59.267 INFO:teuthology.orchestra.run.smithi023.stdout: running: 1 2024-03-24T05:30:59.267 INFO:teuthology.orchestra.run.smithi023.stdout: size: 1 2024-03-24T05:30:59.492 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-mon[25869]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:59.492 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 podman[56630]: 2024-03-24 05:30:59.190841745 +0000 UTC m=+2.723169108 container create c154e31f791c603583c39ad55e50bcdc187a9cbfdd2a089c01a3a8e1166753b5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T05:30:59.492 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 podman[56630]: 2024-03-24 05:30:59.117810693 +0000 UTC m=+2.650138060 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-24T05:30:59.803 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:30:59 smithi063 ceph-mon[30595]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:59.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:30:59 smithi143 ceph-mon[30288]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:30:59.830 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 podman[56630]: 2024-03-24 05:30:59.493816145 +0000 UTC m=+3.026143506 container init c154e31f791c603583c39ad55e50bcdc187a9cbfdd2a089c01a3a8e1166753b5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T05:30:59.831 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 podman[56630]: 2024-03-24 05:30:59.499242929 +0000 UTC m=+3.031570298 container start c154e31f791c603583c39ad55e50bcdc187a9cbfdd2a089c01a3a8e1166753b5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T05:30:59.831 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.503Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-24T05:30:59.831 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.503Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-24T05:30:59.831 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.503Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-24T05:30:59.831 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.503Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-24T05:30:59.831 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.503Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-24T05:30:59.831 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-24T05:30:59.831 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-24T05:30:59.831 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=arp 2024-03-24T05:30:59.831 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-24T05:30:59.832 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-24T05:30:59.832 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-24T05:30:59.832 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-24T05:30:59.832 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-24T05:30:59.832 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-24T05:30:59.832 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-24T05:30:59.832 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-24T05:30:59.832 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=edac 2024-03-24T05:30:59.832 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-24T05:30:59.832 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-24T05:30:59.833 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-24T05:30:59.833 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-24T05:30:59.833 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-24T05:30:59.833 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-24T05:30:59.833 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-24T05:30:59.833 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-24T05:30:59.833 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-24T05:30:59.833 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-24T05:30:59.833 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-24T05:30:59.833 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-24T05:30:59.833 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-24T05:30:59.833 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-24T05:30:59.834 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-24T05:30:59.834 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-24T05:30:59.834 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=os 2024-03-24T05:30:59.834 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-24T05:30:59.834 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-24T05:30:59.834 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-24T05:30:59.834 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-24T05:30:59.834 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-24T05:30:59.834 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.504Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-24T05:30:59.834 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.505Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-24T05:30:59.835 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.505Z caller=node_exporter.go:117 level=info collector=stat 2024-03-24T05:30:59.835 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.505Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-24T05:30:59.835 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.505Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-24T05:30:59.835 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.505Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-24T05:30:59.835 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.505Z caller=node_exporter.go:117 level=info collector=time 2024-03-24T05:30:59.835 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.505Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-24T05:30:59.835 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.505Z caller=node_exporter.go:117 level=info collector=uname 2024-03-24T05:30:59.835 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.505Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-24T05:30:59.835 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.505Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-24T05:30:59.835 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.505Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-24T05:30:59.835 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.505Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-24T05:30:59.835 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a[56856]: ts=2024-03-24T05:30:59.505Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-24T05:30:59.836 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 bash[56630]: c154e31f791c603583c39ad55e50bcdc187a9cbfdd2a089c01a3a8e1166753b5 2024-03-24T05:30:59.836 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:30:59 smithi023 systemd[1]: Started Ceph node-exporter.a for e2aa55c2-e99e-11ee-95cd-87774f69a715. 2024-03-24T05:31:00.031 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-24T05:31:00.271 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T05:31:00.943 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:31:00 smithi023 ceph-mon[25869]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-24T05:31:00.943 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:31:00 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:31:00.943 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:31:00 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:31:00.943 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:31:00 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:31:00.943 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:31:00 smithi023 ceph-mon[25869]: Deploying daemon node-exporter.b on smithi063 2024-03-24T05:31:00.943 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:31:00 smithi023 ceph-mon[25869]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:31:01.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:31:00 smithi063 ceph-mon[30595]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-24T05:31:01.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:31:00 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:31:01.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:31:00 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:31:01.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:31:00 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:31:01.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:31:00 smithi063 ceph-mon[30595]: Deploying daemon node-exporter.b on smithi063 2024-03-24T05:31:01.054 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:31:00 smithi063 ceph-mon[30595]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:31:01.055 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:00 smithi063 bash[43906]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-24T05:31:01.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:31:00 smithi143 ceph-mon[30288]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-24T05:31:01.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:31:00 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:31:01.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:31:00 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:31:01.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:31:00 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:31:01.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:31:00 smithi143 ceph-mon[30288]: Deploying daemon node-exporter.b on smithi063 2024-03-24T05:31:01.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:31:00 smithi143 ceph-mon[30288]: from='mgr.14448 172.21.15.23:0/2070198498' entity='mgr.y' 2024-03-24T05:31:01.553 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:01 smithi063 bash[43906]: Getting image source signatures 2024-03-24T05:31:01.554 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:01 smithi063 bash[43906]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-24T05:31:01.554 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:01 smithi063 bash[43906]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-24T05:31:01.554 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:01 smithi063 bash[43906]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-24T05:31:01.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:31:01 smithi023 ceph-mon[25869]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:31:01.953 INFO:teuthology.orchestra.run.smithi023.stdout:osd 3 40s ago - 2024-03-24T05:31:02.053 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:31:01 smithi063 ceph-mon[30595]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:31:02.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:31:01 smithi143 ceph-mon[30288]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T05:31:02.491 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-24T05:31:02.501 INFO:tasks.cephadm:Teardown begin 2024-03-24T05:31:02.501 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T05:31:02.530 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T05:31:02.560 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T05:31:02.595 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-24T05:31:02.595 DEBUG:teuthology.orchestra.run.smithi023:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T05:31:02.609 DEBUG:teuthology.orchestra.run.smithi063:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T05:31:02.623 DEBUG:teuthology.orchestra.run.smithi143:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T05:31:02.651 INFO:tasks.cephadm:Stopping all daemons... 2024-03-24T05:31:02.652 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-24T05:31:02.652 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mon.a 2024-03-24T05:31:02.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:31:02 smithi023 systemd[1]: Stopping Ceph mon.a for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:31:02.985 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:02 smithi063 bash[43906]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-24T05:31:02.985 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:02 smithi063 bash[43906]: Writing manifest to image destination 2024-03-24T05:31:03.218 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:31:02 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-a[25846]: 2024-03-24T05:31:02.964+0000 7f08870f1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T05:31:03.219 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:31:02 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-a[25846]: 2024-03-24T05:31:02.964+0000 7f08870f1700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-24T05:31:03.219 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:31:02 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: [24/Mar/2024:05:31:02] ENGINE Bus STOPPING 2024-03-24T05:31:03.219 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:31:02 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: [24/Mar/2024:05:31:02] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T05:31:03.219 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:31:02 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: [24/Mar/2024:05:31:02] ENGINE Bus STOPPED 2024-03-24T05:31:03.219 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:31:02 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: [24/Mar/2024:05:31:02] ENGINE Bus STARTING 2024-03-24T05:31:03.219 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:31:03 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: [24/Mar/2024:05:31:03] ENGINE Serving on http://:::9283 2024-03-24T05:31:03.219 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:31:03 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: [24/Mar/2024:05:31:03] ENGINE Bus STARTED 2024-03-24T05:31:03.240 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:02 smithi063 podman[43906]: 2024-03-24 05:31:02.97789426 +0000 UTC m=+2.213219325 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-24T05:31:03.240 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 podman[43906]: 2024-03-24 05:31:03.060016109 +0000 UTC m=+2.295341213 container create 050fa71e9d3405c953209f4caba2d060d46fe944df7264c47abc3d813145e11b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T05:31:03.240 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 podman[43906]: 2024-03-24 05:31:03.241723513 +0000 UTC m=+2.477048595 container init 050fa71e9d3405c953209f4caba2d060d46fe944df7264c47abc3d813145e11b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T05:31:03.553 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:31:03 smithi023 podman[57190]: 2024-03-24 05:31:03.220207297 +0000 UTC m=+0.381593483 container died 8e5cbfa90e96dc6d5e9350efa4ff75eece07fa707c50e1a69e98038ec8578f97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-a, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-24T05:31:03.554 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 podman[43906]: 2024-03-24 05:31:03.247363705 +0000 UTC m=+2.482688776 container start 050fa71e9d3405c953209f4caba2d060d46fe944df7264c47abc3d813145e11b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T05:31:03.554 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.251Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-24T05:31:03.554 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.251Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-24T05:31:03.554 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.251Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-24T05:31:03.554 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.251Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-24T05:31:03.554 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-24T05:31:03.554 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-24T05:31:03.554 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-24T05:31:03.554 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=arp 2024-03-24T05:31:03.554 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-24T05:31:03.554 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-24T05:31:03.554 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-24T05:31:03.555 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-24T05:31:03.555 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-24T05:31:03.555 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-24T05:31:03.555 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-24T05:31:03.555 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-24T05:31:03.555 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=edac 2024-03-24T05:31:03.555 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-24T05:31:03.555 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-24T05:31:03.555 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-24T05:31:03.556 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-24T05:31:03.556 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-24T05:31:03.556 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-24T05:31:03.556 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-24T05:31:03.556 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-24T05:31:03.556 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-24T05:31:03.556 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-24T05:31:03.556 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-24T05:31:03.556 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-24T05:31:03.556 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-24T05:31:03.556 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-24T05:31:03.556 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-24T05:31:03.556 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-24T05:31:03.556 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=os 2024-03-24T05:31:03.556 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-24T05:31:03.557 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-24T05:31:03.557 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-24T05:31:03.557 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-24T05:31:03.557 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-24T05:31:03.557 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-24T05:31:03.557 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-24T05:31:03.557 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=stat 2024-03-24T05:31:03.557 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-24T05:31:03.557 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-24T05:31:03.557 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-24T05:31:03.557 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=time 2024-03-24T05:31:03.557 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-24T05:31:03.557 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=uname 2024-03-24T05:31:03.557 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-24T05:31:03.557 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.252Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-24T05:31:03.557 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.253Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-24T05:31:03.558 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.253Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-24T05:31:03.558 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b[43991]: ts=2024-03-24T05:31:03.253Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-24T05:31:03.558 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 bash[43906]: 050fa71e9d3405c953209f4caba2d060d46fe944df7264c47abc3d813145e11b 2024-03-24T05:31:03.558 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:03 smithi063 systemd[1]: Started Ceph node-exporter.b for e2aa55c2-e99e-11ee-95cd-87774f69a715. 2024-03-24T05:31:03.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:31:03 smithi023 podman[57190]: 2024-03-24 05:31:03.70366597 +0000 UTC m=+0.865052152 container cleanup 8e5cbfa90e96dc6d5e9350efa4ff75eece07fa707c50e1a69e98038ec8578f97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, ceph=True) 2024-03-24T05:31:03.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:31:03 smithi023 bash[57190]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-a 2024-03-24T05:31:04.142 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 05:31:03 smithi023 podman[57220]: 2024-03-24 05:31:03.971685237 +0000 UTC m=+0.747868287 container remove 8e5cbfa90e96dc6d5e9350efa4ff75eece07fa707c50e1a69e98038ec8578f97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-a, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T05:31:04.158 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mon.a.service' 2024-03-24T05:31:04.234 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T05:31:04.234 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-24T05:31:04.234 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-24T05:31:04.234 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mon.b 2024-03-24T05:31:04.554 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:31:04 smithi063 systemd[1]: Stopping Ceph mon.b for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:31:04.884 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:31:04 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-b[30572]: 2024-03-24T05:31:04.558+0000 7f64207ff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T05:31:04.884 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:31:04 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-b[30572]: 2024-03-24T05:31:04.558+0000 7f64207ff700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-24T05:31:04.885 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:31:04 smithi063 podman[44068]: 2024-03-24 05:31:04.80487634 +0000 UTC m=+0.369287087 container died 055378e48274efe9de3dcb07b05f5d9cd3bbf6e37676a4a64815252af62365f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-b, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T05:31:05.504 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:31:05 smithi063 podman[44068]: 2024-03-24 05:31:05.240387302 +0000 UTC m=+0.804798057 container cleanup 055378e48274efe9de3dcb07b05f5d9cd3bbf6e37676a4a64815252af62365f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-b, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b) 2024-03-24T05:31:05.504 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:31:05 smithi063 bash[44068]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-b 2024-03-24T05:31:05.504 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 24 05:31:05 smithi063 podman[44088]: 2024-03-24 05:31:05.42510102 +0000 UTC m=+0.614712093 container remove 055378e48274efe9de3dcb07b05f5d9cd3bbf6e37676a4a64815252af62365f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-24T05:31:05.675 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mon.b.service' 2024-03-24T05:31:05.723 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T05:31:05.724 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-24T05:31:05.724 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-24T05:31:05.724 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mon.c 2024-03-24T05:31:06.027 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:31:05 smithi143 systemd[1]: Stopping Ceph mon.c for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:31:06.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:31:06 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-c[30265]: 2024-03-24T05:31:06.026+0000 7f5668671700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T05:31:06.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:31:06 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-c[30265]: 2024-03-24T05:31:06.026+0000 7f5668671700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-24T05:31:06.308 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:31:06 smithi143 podman[43428]: 2024-03-24 05:31:06.088776085 +0000 UTC m=+0.165216643 container died 28c1a290e01f7ae2a92ff3b85ef78f1b963c2f8c5e15383dde992b53fb66380a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS) 2024-03-24T05:31:06.807 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:31:06 smithi143 podman[43428]: 2024-03-24 05:31:06.540738808 +0000 UTC m=+0.617179444 container cleanup 28c1a290e01f7ae2a92ff3b85ef78f1b963c2f8c5e15383dde992b53fb66380a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-c, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-24T05:31:06.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 05:31:06 smithi143 bash[43428]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mon-c 2024-03-24T05:31:06.936 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mon.c.service' 2024-03-24T05:31:06.981 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T05:31:06.982 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-24T05:31:06.982 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-24T05:31:06.982 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mgr.y 2024-03-24T05:31:07.304 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:31:07 smithi023 systemd[1]: Stopping Ceph mgr.y for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:31:07.579 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:31:07 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y[26129]: 2024-03-24T05:31:07.304+0000 7f016f1e1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T05:31:07.580 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:31:07 smithi023 podman[57343]: 2024-03-24 05:31:07.335091856 +0000 UTC m=+0.153821118 container died 877f2bae6dcbd50a250036be842e1b30a933533a0f7102b008752e10180bd804 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-03-24T05:31:08.052 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:31:07 smithi023 podman[57343]: 2024-03-24 05:31:07.8097828 +0000 UTC m=+0.628512056 container cleanup 877f2bae6dcbd50a250036be842e1b30a933533a0f7102b008752e10180bd804 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-24T05:31:08.052 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 24 05:31:07 smithi023 bash[57343]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-y 2024-03-24T05:31:08.191 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-24T05:31:08.228 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T05:31:08.229 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-24T05:31:08.229 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-24T05:31:08.229 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mgr.x 2024-03-24T05:31:08.553 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:31:08 smithi063 systemd[1]: Stopping Ceph mgr.x for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:31:08.837 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:31:08 smithi063 podman[44198]: 2024-03-24 05:31:08.561796357 +0000 UTC m=+0.133500954 container died d03b804431107a5ebd5b6da9cbfa74e1a28dafaf7f869d0505e5c865d0f44514 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x, RELEASE=squid-db0330b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-24T05:31:09.261 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:31:08 smithi063 podman[44198]: 2024-03-24 05:31:08.904198996 +0000 UTC m=+0.475903593 container cleanup d03b804431107a5ebd5b6da9cbfa74e1a28dafaf7f869d0505e5c865d0f44514 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-24T05:31:09.261 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:31:08 smithi063 bash[44198]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x 2024-03-24T05:31:09.261 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:31:09 smithi063 podman[44217]: 2024-03-24 05:31:09.063807447 +0000 UTC m=+0.494596481 container remove d03b804431107a5ebd5b6da9cbfa74e1a28dafaf7f869d0505e5c865d0f44514 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-x, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-24T05:31:09.261 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Mar 24 05:31:09 smithi063 systemd[1]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-24T05:31:09.279 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-24T05:31:09.315 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T05:31:09.316 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-24T05:31:09.316 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-24T05:31:09.316 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mgr.z 2024-03-24T05:31:09.617 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:31:09 smithi143 systemd[1]: Stopping Ceph mgr.z for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:31:09.869 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:31:09 smithi143 podman[43559]: 2024-03-24 05:31:09.618967104 +0000 UTC m=+0.113370088 container died 47dd699bae4bccc00e44ffa7d308c0dd1cebc7b6c3924f69c29fb30b06b15433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-24T05:31:10.144 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:31:09 smithi143 podman[43559]: 2024-03-24 05:31:09.961448176 +0000 UTC m=+0.455851168 container cleanup 47dd699bae4bccc00e44ffa7d308c0dd1cebc7b6c3924f69c29fb30b06b15433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.build-date=20240307) 2024-03-24T05:31:10.145 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:31:09 smithi143 bash[43559]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z 2024-03-24T05:31:10.377 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mgr.z.service' 2024-03-24T05:31:10.401 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:31:10 smithi143 podman[43578]: 2024-03-24 05:31:10.146080115 +0000 UTC m=+0.523590471 container remove 47dd699bae4bccc00e44ffa7d308c0dd1cebc7b6c3924f69c29fb30b06b15433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-mgr-z, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True) 2024-03-24T05:31:10.401 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:31:10 smithi143 systemd[1]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-03-24T05:31:10.401 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:31:10 smithi143 systemd[1]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mgr.z.service: Failed with result 'exit-code'. 2024-03-24T05:31:10.401 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:31:10 smithi143 systemd[1]: Stopped Ceph mgr.z for e2aa55c2-e99e-11ee-95cd-87774f69a715. 2024-03-24T05:31:10.401 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 05:31:10 smithi143 systemd[1]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@mgr.z.service: Consumed 16.323s CPU time. 2024-03-24T05:31:10.413 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T05:31:10.413 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-24T05:31:10.413 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-24T05:31:10.413 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@osd.0 2024-03-24T05:31:10.715 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:31:10 smithi023 systemd[1]: Stopping Ceph osd.0 for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:31:11.080 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:31:10 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-0[40745]: 2024-03-24T05:31:10.716+0000 7ff593aa7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T05:31:11.080 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:31:10 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-0[40745]: 2024-03-24T05:31:10.716+0000 7ff593aa7700 -1 osd.0 31 *** Got signal Terminated *** 2024-03-24T05:31:11.080 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:31:10 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-0[40745]: 2024-03-24T05:31:10.716+0000 7ff593aa7700 -1 osd.0 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T05:31:11.803 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:31:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:31:11.533Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.23:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.23:8765: connect: connection refused" 2024-03-24T05:31:11.804 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:31:11 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:31:11.533Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.23:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.23:8765: connect: connection refused" 2024-03-24T05:31:16.330 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:31:16 smithi023 podman[57477]: 2024-03-24 05:31:16.041127079 +0000 UTC m=+5.435734936 container died 1e2642d25dd6d5bd09d0955e50eb465e140f8f4e737b5a2b1446f645c7f103a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-0, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-24T05:31:16.754 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:31:16 smithi023 podman[57477]: 2024-03-24 05:31:16.521262729 +0000 UTC m=+5.915870589 container cleanup 1e2642d25dd6d5bd09d0955e50eb465e140f8f4e737b5a2b1446f645c7f103a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-24T05:31:16.754 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:31:16 smithi023 bash[57477]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-0 2024-03-24T05:31:17.064 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:31:16 smithi023 podman[57499]: 2024-03-24 05:31:16.755793441 +0000 UTC m=+0.717046737 container remove 1e2642d25dd6d5bd09d0955e50eb465e140f8f4e737b5a2b1446f645c7f103a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-24T05:31:17.330 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:31:17 smithi023 podman[57596]: 2024-03-24 05:31:17.064944685 +0000 UTC m=+0.111555696 container create 0b8c4e97e9cceecd11d5388c6f81f486f1b74ade3b4acbe5658263fedd300361 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-db0330b, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-24T05:31:17.331 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:31:17 smithi023 podman[57596]: 2024-03-24 05:31:16.971042138 +0000 UTC m=+0.017653172 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T05:31:17.331 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:31:17 smithi023 podman[57596]: 2024-03-24 05:31:17.276213821 +0000 UTC m=+0.322824846 container init 0b8c4e97e9cceecd11d5388c6f81f486f1b74ade3b4acbe5658263fedd300361 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-24T05:31:17.331 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:31:17 smithi023 podman[57596]: 2024-03-24 05:31:17.281848666 +0000 UTC m=+0.328459677 container start 0b8c4e97e9cceecd11d5388c6f81f486f1b74ade3b4acbe5658263fedd300361 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T05:31:17.609 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:31:17 smithi023 podman[57596]: 2024-03-24 05:31:17.3504464 +0000 UTC m=+0.397057426 container attach 0b8c4e97e9cceecd11d5388c6f81f486f1b74ade3b4acbe5658263fedd300361 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.build-date=20240307) 2024-03-24T05:31:17.609 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:31:17 smithi023 podman[57596]: 2024-03-24 05:31:17.583059708 +0000 UTC m=+0.629670725 container died 0b8c4e97e9cceecd11d5388c6f81f486f1b74ade3b4acbe5658263fedd300361 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T05:31:18.116 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@osd.0.service' 2024-03-24T05:31:18.141 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:31:18 smithi023 podman[57642]: 2024-03-24 05:31:18.095213034 +0000 UTC m=+0.506845412 container remove 0b8c4e97e9cceecd11d5388c6f81f486f1b74ade3b4acbe5658263fedd300361 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T05:31:18.141 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:31:18 smithi023 systemd[1]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-24T05:31:18.142 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:31:18 smithi023 systemd[1]: Stopped Ceph osd.0 for e2aa55c2-e99e-11ee-95cd-87774f69a715. 2024-03-24T05:31:18.142 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 05:31:18 smithi023 systemd[1]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@osd.0.service: Consumed 3.858s CPU time. 2024-03-24T05:31:18.156 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T05:31:18.156 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-24T05:31:18.156 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-24T05:31:18.156 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@osd.1 2024-03-24T05:31:18.553 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:31:18 smithi063 systemd[1]: Stopping Ceph osd.1 for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:31:19.554 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:31:19 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-1[35167]: 2024-03-24T05:31:19.154+0000 7f93c7efa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T05:31:19.554 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:31:19 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-1[35167]: 2024-03-24T05:31:19.154+0000 7f93c7efa700 -1 osd.1 31 *** Got signal Terminated *** 2024-03-24T05:31:19.554 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:31:19 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-1[35167]: 2024-03-24T05:31:19.154+0000 7f93c7efa700 -1 osd.1 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T05:31:24.866 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:31:24 smithi063 podman[44331]: 2024-03-24 05:31:24.576146875 +0000 UTC m=+6.226065033 container died 2022b45651c2bd7090e68d4aac792254e131e78ac11ed7face7d77a30e153ecc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-1, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T05:31:24.866 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:31:24 smithi063 podman[44331]: 2024-03-24 05:31:24.867410155 +0000 UTC m=+6.517328319 container cleanup 2022b45651c2bd7090e68d4aac792254e131e78ac11ed7face7d77a30e153ecc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-1, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-24T05:31:25.303 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:31:24 smithi063 bash[44331]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-1 2024-03-24T05:31:25.304 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:31:25 smithi063 podman[44354]: 2024-03-24 05:31:25.043310141 +0000 UTC m=+0.460821269 container remove 2022b45651c2bd7090e68d4aac792254e131e78ac11ed7face7d77a30e153ecc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-03-24T05:31:25.679 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:31:25 smithi063 podman[44451]: 2024-03-24 05:31:25.26513571 +0000 UTC m=+0.017967641 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T05:31:25.680 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:31:25 smithi063 podman[44451]: 2024-03-24 05:31:25.419546618 +0000 UTC m=+0.172378589 container create f5c441538b90bd489e6ded9a0bdb4403e8e29a244876e23b5466fb91a354bbe0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b) 2024-03-24T05:31:25.680 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:31:25 smithi063 podman[44451]: 2024-03-24 05:31:25.596795499 +0000 UTC m=+0.349627434 container init f5c441538b90bd489e6ded9a0bdb4403e8e29a244876e23b5466fb91a354bbe0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T05:31:25.680 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:31:25 smithi063 podman[44451]: 2024-03-24 05:31:25.601789223 +0000 UTC m=+0.354621148 container start f5c441538b90bd489e6ded9a0bdb4403e8e29a244876e23b5466fb91a354bbe0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-24T05:31:26.053 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:31:25 smithi063 podman[44451]: 2024-03-24 05:31:25.680198787 +0000 UTC m=+0.433030713 container attach f5c441538b90bd489e6ded9a0bdb4403e8e29a244876e23b5466fb91a354bbe0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-24T05:31:26.054 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 24 05:31:25 smithi063 podman[44451]: 2024-03-24 05:31:25.899688417 +0000 UTC m=+0.652520353 container died f5c441538b90bd489e6ded9a0bdb4403e8e29a244876e23b5466fb91a354bbe0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-1-deactivate, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-24T05:31:26.455 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@osd.1.service' 2024-03-24T05:31:26.496 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T05:31:26.496 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-24T05:31:26.496 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-24T05:31:26.497 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@osd.2 2024-03-24T05:31:26.806 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:31:26 smithi143 systemd[1]: Stopping Ceph osd.2 for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:31:27.057 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:31:26 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-2[35531]: 2024-03-24T05:31:26.806+0000 7ff89deee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T05:31:27.058 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:31:26 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-2[35531]: 2024-03-24T05:31:26.806+0000 7ff89deee700 -1 osd.2 31 *** Got signal Terminated *** 2024-03-24T05:31:27.058 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:31:26 smithi143 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-2[35531]: 2024-03-24T05:31:26.806+0000 7ff89deee700 -1 osd.2 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T05:31:32.455 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:31:32 smithi143 podman[43695]: 2024-03-24 05:31:32.177128715 +0000 UTC m=+5.480623599 container died e3de23f43064cbada5b361413aed3a440dd783b4a85d369c542ab56c698acb99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-2, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T05:31:32.787 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:31:32 smithi143 podman[43695]: 2024-03-24 05:31:32.554427013 +0000 UTC m=+5.857921905 container cleanup e3de23f43064cbada5b361413aed3a440dd783b4a85d369c542ab56c698acb99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-24T05:31:32.787 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:31:32 smithi143 bash[43695]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-2 2024-03-24T05:31:33.057 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:31:32 smithi143 podman[43729]: 2024-03-24 05:31:32.788843848 +0000 UTC m=+0.605358195 container remove e3de23f43064cbada5b361413aed3a440dd783b4a85d369c542ab56c698acb99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-24T05:31:33.374 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:31:33 smithi143 podman[43825]: 2024-03-24 05:31:33.081386469 +0000 UTC m=+0.098311191 container create 09b4e6a1214911b7febf42bd6cd6e45741c4de785641bc8fdd706cf608e4af90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T05:31:33.374 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:31:33 smithi143 podman[43825]: 2024-03-24 05:31:33.001261116 +0000 UTC m=+0.018185808 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T05:31:33.374 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:31:33 smithi143 podman[43825]: 2024-03-24 05:31:33.292326083 +0000 UTC m=+0.309250776 container init 09b4e6a1214911b7febf42bd6cd6e45741c4de785641bc8fdd706cf608e4af90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-24T05:31:33.375 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:31:33 smithi143 podman[43825]: 2024-03-24 05:31:33.298176491 +0000 UTC m=+0.315101180 container start 09b4e6a1214911b7febf42bd6cd6e45741c4de785641bc8fdd706cf608e4af90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-24T05:31:33.375 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:31:33 smithi143 podman[43825]: 2024-03-24 05:31:33.375358603 +0000 UTC m=+0.392283301 container attach 09b4e6a1214911b7febf42bd6cd6e45741c4de785641bc8fdd706cf608e4af90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-2-deactivate, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T05:31:33.807 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 05:31:33 smithi143 podman[43825]: 2024-03-24 05:31:33.590015151 +0000 UTC m=+0.606939833 container died 09b4e6a1214911b7febf42bd6cd6e45741c4de785641bc8fdd706cf608e4af90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, ceph=True) 2024-03-24T05:31:34.215 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@osd.2.service' 2024-03-24T05:31:34.256 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T05:31:34.256 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-24T05:31:34.256 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-24T05:31:34.256 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@rgw.foo.a 2024-03-24T05:31:34.550 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 24 05:31:34 smithi023 systemd[1]: Stopping Ceph rgw.foo.a for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:31:34.829 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 24 05:31:34 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-rgw-foo-a[47686]: 2024-03-24T05:31:34.550+0000 7f20d6cb8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T05:31:34.829 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 24 05:31:34 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-rgw-foo-a[47686]: 2024-03-24T05:31:34.550+0000 7f20e5528ac0 -1 shutting down 2024-03-24T05:31:44.830 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 24 05:31:44 smithi023 bash[57729]: time="2024-03-24T05:31:44Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-24T05:31:44.830 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 24 05:31:44 smithi023 podman[57729]: 2024-03-24 05:31:44.580708821 +0000 UTC m=+10.128546370 container died 29ebffacb9c38b8a5850a171f9641d1a000e1260dd8a7c227b37d699bd9c45df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-rgw-foo-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, ceph=True) 2024-03-24T05:31:45.196 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 24 05:31:45 smithi023 podman[57729]: 2024-03-24 05:31:45.038686162 +0000 UTC m=+10.586523758 container cleanup 29ebffacb9c38b8a5850a171f9641d1a000e1260dd8a7c227b37d699bd9c45df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-24T05:31:45.196 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 24 05:31:45 smithi023 bash[57729]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-rgw-foo-a 2024-03-24T05:31:45.579 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 24 05:31:45 smithi023 podman[57766]: 2024-03-24 05:31:45.415005649 +0000 UTC m=+0.831534324 container remove 29ebffacb9c38b8a5850a171f9641d1a000e1260dd8a7c227b37d699bd9c45df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-rgw-foo-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T05:31:45.580 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 24 05:31:45 smithi023 systemd[1]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-24T05:31:45.606 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@rgw.foo.a.service' 2024-03-24T05:31:45.644 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T05:31:45.644 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-24T05:31:45.644 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-24T05:31:45.644 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@prometheus.a 2024-03-24T05:31:45.952 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:31:45 smithi063 systemd[1]: Stopping Ceph prometheus.a for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:31:46.242 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:31:45 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:31:45.953Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-24T05:31:46.242 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:31:45 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:31:45.954Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-24T05:31:46.242 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:31:45 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:31:45.954Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-24T05:31:46.242 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:31:45 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:31:45.954Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-24T05:31:46.242 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:31:45 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:31:45.954Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-24T05:31:46.243 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:31:45 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:31:45.954Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-24T05:31:46.243 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:31:45 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:31:45.954Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-24T05:31:46.243 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:31:45 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:31:45.954Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-24T05:31:46.243 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:31:45 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:31:45.954Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-24T05:31:46.243 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:31:45 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:31:45.954Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-24T05:31:46.243 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:31:45 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:31:45.954Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-24T05:31:46.243 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:31:45 smithi063 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a[41756]: ts=2024-03-24T05:31:45.954Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-24T05:31:46.243 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:31:45 smithi063 podman[44594]: 2024-03-24 05:31:45.964187018 +0000 UTC m=+0.112929658 container died d84f0a1ee3acc8ba8bf428ba2438e998ae74fa4aab84e906118de5695e424a51 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T05:31:46.554 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:31:46 smithi063 podman[44594]: 2024-03-24 05:31:46.350741288 +0000 UTC m=+0.499483925 container cleanup d84f0a1ee3acc8ba8bf428ba2438e998ae74fa4aab84e906118de5695e424a51 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T05:31:46.554 INFO:journalctl@ceph.prometheus.a.smithi063.stdout:Mar 24 05:31:46 smithi063 bash[44594]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-prometheus-a 2024-03-24T05:31:46.729 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@prometheus.a.service' 2024-03-24T05:31:46.766 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T05:31:46.766 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-24T05:31:46.767 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T05:31:46.962 INFO:teuthology.orchestra.run.smithi023.stdout:Deleting cluster with fsid: e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:31:48.053 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 24 05:31:47 smithi023 systemd[1]: Stopping Ceph alertmanager.a for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:31:48.329 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 24 05:31:48 smithi023 ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-alertmanager-a[55086]: ts=2024-03-24T05:31:48.054Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-24T05:31:48.329 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 24 05:31:48 smithi023 podman[58084]: 2024-03-24 05:31:48.085125407 +0000 UTC m=+0.139383797 container died 5926e418c7d24b030e79c75a1bfae3a971d92aee7f36c547ec6e7df84f835226 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T05:31:48.806 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 24 05:31:48 smithi023 podman[58084]: 2024-03-24 05:31:48.409808888 +0000 UTC m=+0.464067319 container cleanup 5926e418c7d24b030e79c75a1bfae3a971d92aee7f36c547ec6e7df84f835226 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T05:31:48.806 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 24 05:31:48 smithi023 bash[58084]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-alertmanager-a 2024-03-24T05:31:48.806 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 24 05:31:48 smithi023 podman[58103]: 2024-03-24 05:31:48.54437978 +0000 UTC m=+0.459917903 container remove 5926e418c7d24b030e79c75a1bfae3a971d92aee7f36c547ec6e7df84f835226 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T05:31:48.806 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 24 05:31:48 smithi023 podman[58103]: 2024-03-24 05:31:48.603160288 +0000 UTC m=+0.518698410 volume remove 4ff0fb7004ebb986ecb013472b3e914cbba5427155cc838c8fdde9e05455219b 2024-03-24T05:31:49.079 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:31:49 smithi023 systemd[1]: Stopping Ceph node-exporter.a for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:31:49.080 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 24 05:31:48 smithi023 systemd[1]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-24T05:31:49.080 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 24 05:31:48 smithi023 systemd[1]: Stopped Ceph alertmanager.a for e2aa55c2-e99e-11ee-95cd-87774f69a715. 2024-03-24T05:31:49.505 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:31:49 smithi023 podman[58221]: 2024-03-24 05:31:49.252706089 +0000 UTC m=+0.095625094 container died c154e31f791c603583c39ad55e50bcdc187a9cbfdd2a089c01a3a8e1166753b5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T05:31:49.829 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:31:49 smithi023 podman[58221]: 2024-03-24 05:31:49.664187387 +0000 UTC m=+0.507106440 container cleanup c154e31f791c603583c39ad55e50bcdc187a9cbfdd2a089c01a3a8e1166753b5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T05:31:49.829 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:31:49 smithi023 bash[58221]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a 2024-03-24T05:31:50.191 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:31:49 smithi023 podman[58239]: 2024-03-24 05:31:49.898717748 +0000 UTC m=+0.647354500 container remove c154e31f791c603583c39ad55e50bcdc187a9cbfdd2a089c01a3a8e1166753b5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T05:31:50.192 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:31:49 smithi023 systemd[1]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-24T05:31:50.192 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:31:50 smithi023 systemd[1]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-24T05:31:50.192 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:31:50 smithi023 systemd[1]: Stopped Ceph node-exporter.a for e2aa55c2-e99e-11ee-95cd-87774f69a715. 2024-03-24T05:31:50.192 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 24 05:31:50 smithi023 systemd[1]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@node-exporter.a.service: Consumed 1.757s CPU time. 2024-03-24T05:31:50.665 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T05:31:50.857 INFO:teuthology.orchestra.run.smithi063.stdout:Deleting cluster with fsid: e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:31:51.804 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:31:51 smithi063 systemd[1]: Stopping Ceph grafana.a for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:31:52.221 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:31:51 smithi063 podman[44931]: 2024-03-24 05:31:51.952313133 +0000 UTC m=+0.124502154 container died 82dd7fc3b52449c2c9edcb0a34943137aa3f24ccdd6d23c5766a244392aec709 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a, vcs-type=git, maintainer=Paul Cuzner , name=ubi8, release=239.1651231664, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, io.openshift.expose-services=, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0) 2024-03-24T05:31:52.554 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:31:52 smithi063 podman[44931]: 2024-03-24 05:31:52.304825927 +0000 UTC m=+0.477014948 container cleanup 82dd7fc3b52449c2c9edcb0a34943137aa3f24ccdd6d23c5766a244392aec709 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, architecture=x86_64, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., release=239.1651231664, io.buildah.version=1.32.0, maintainer=Paul Cuzner , io.openshift.expose-services=, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, distribution-scope=public) 2024-03-24T05:31:52.554 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:31:52 smithi063 bash[44931]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a 2024-03-24T05:31:52.898 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:31:52 smithi063 podman[44950]: 2024-03-24 05:31:52.555805531 +0000 UTC m=+0.606007863 container remove 82dd7fc3b52449c2c9edcb0a34943137aa3f24ccdd6d23c5766a244392aec709 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-grafana-a, distribution-scope=public, io.buildah.version=1.32.0, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , architecture=x86_64, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-04-29T11:48:11.055897, io.openshift.tags=base rhel8, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, com.redhat.component=ubi8-container) 2024-03-24T05:31:52.898 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:31:52 smithi063 systemd[1]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-24T05:31:52.898 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:31:52 smithi063 systemd[1]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-24T05:31:52.898 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:31:52 smithi063 systemd[1]: Stopped Ceph grafana.a for e2aa55c2-e99e-11ee-95cd-87774f69a715. 2024-03-24T05:31:52.898 INFO:journalctl@ceph.grafana.a.smithi063.stdout:Mar 24 05:31:52 smithi063 systemd[1]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@grafana.a.service: Consumed 2.771s CPU time. 2024-03-24T05:31:53.154 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:52 smithi063 systemd[1]: Stopping Ceph node-exporter.b for e2aa55c2-e99e-11ee-95cd-87774f69a715... 2024-03-24T05:31:53.541 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:53 smithi063 podman[45069]: 2024-03-24 05:31:53.15563352 +0000 UTC m=+0.117170716 container died 050fa71e9d3405c953209f4caba2d060d46fe944df7264c47abc3d813145e11b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T05:31:53.803 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:53 smithi063 podman[45069]: 2024-03-24 05:31:53.67492165 +0000 UTC m=+0.636458853 container cleanup 050fa71e9d3405c953209f4caba2d060d46fe944df7264c47abc3d813145e11b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T05:31:53.803 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:53 smithi063 bash[45069]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b 2024-03-24T05:31:54.253 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:53 smithi063 podman[45088]: 2024-03-24 05:31:53.884303693 +0000 UTC m=+0.730363031 container remove 050fa71e9d3405c953209f4caba2d060d46fe944df7264c47abc3d813145e11b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T05:31:54.253 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:53 smithi063 systemd[1]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-24T05:31:54.253 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:54 smithi063 systemd[1]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-24T05:31:54.253 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:54 smithi063 systemd[1]: Stopped Ceph node-exporter.b for e2aa55c2-e99e-11ee-95cd-87774f69a715. 2024-03-24T05:31:54.253 INFO:journalctl@ceph.node-exporter.b.smithi063.stdout:Mar 24 05:31:54 smithi063 systemd[1]: ceph-e2aa55c2-e99e-11ee-95cd-87774f69a715@node-exporter.b.service: Consumed 1.719s CPU time. 2024-03-24T05:31:54.799 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T05:31:54.991 INFO:teuthology.orchestra.run.smithi143.stdout:Deleting cluster with fsid: e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:31:56.029 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T05:31:56.064 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T05:31:56.093 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T05:31:56.122 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-24T05:31:56.124 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617855/remote/smithi023/crash 2024-03-24T05:31:56.125 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/crash -- . 2024-03-24T05:31:56.174 INFO:teuthology.orchestra.run.smithi023.stderr:tar: /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T05:31:56.174 INFO:teuthology.orchestra.run.smithi023.stderr:tar: Error is not recoverable: exiting now 2024-03-24T05:31:56.176 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617855/remote/smithi063/crash 2024-03-24T05:31:56.177 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/crash -- . 2024-03-24T05:31:56.216 INFO:teuthology.orchestra.run.smithi063.stderr:tar: /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T05:31:56.216 INFO:teuthology.orchestra.run.smithi063.stderr:tar: Error is not recoverable: exiting now 2024-03-24T05:31:56.218 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617855/remote/smithi143/crash 2024-03-24T05:31:56.219 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/crash -- . 2024-03-24T05:31:56.261 INFO:teuthology.orchestra.run.smithi143.stderr:tar: /var/lib/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T05:31:56.261 INFO:teuthology.orchestra.run.smithi143.stderr:tar: Error is not recoverable: exiting now 2024-03-24T05:31:56.262 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-24T05:31:56.262 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T05:31:56.295 INFO:teuthology.orchestra.run.smithi023.stdout:2024-03-24T05:28:10.713557+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T05:31:56.295 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-24T05:31:56.295 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[SEC\]' /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T05:31:56.367 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[ERR\]' /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T05:31:56.438 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[WRN\]' /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T05:31:56.509 INFO:teuthology.orchestra.run.smithi023.stdout:2024-03-24T05:28:10.713557+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T05:31:56.509 INFO:tasks.cephadm:Compressing logs... 2024-03-24T05:31:56.509 DEBUG:teuthology.orchestra.run.smithi023:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T05:31:56.552 DEBUG:teuthology.orchestra.run.smithi063:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T05:31:56.554 DEBUG:teuthology.orchestra.run.smithi143:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T05:31:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T05:31:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:: No such file or directory 2024-03-24T05:31:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-24T05:31:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T05:31:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-24T05:31:56.583 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-24T05:31:56.583 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-24T05:31:56.583 INFO:teuthology.orchestra.run.smithi023.stderr: 57.6% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-24T05:31:56.583 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T05:31:56.583 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.log 2024-03-24T05:31:56.583 INFO:teuthology.orchestra.run.smithi023.stderr: 89.3% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T05:31:56.584 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T05:31:56.584 INFO:teuthology.orchestra.run.smithi023.stderr: 90.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T05:31:56.584 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.log: 82.8% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T05:31:56.584 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-volume.log: 84.8% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T05:31:56.585 INFO:teuthology.orchestra.run.smithi063.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T05:31:56.585 INFO:teuthology.orchestra.run.smithi063.stderr:: No such file or directory 2024-03-24T05:31:56.585 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T05:31:56.585 INFO:teuthology.orchestra.run.smithi063.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-24T05:31:56.585 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-24T05:31:56.585 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.audit.log: /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T05:31:56.585 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.log 2024-03-24T05:31:56.585 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose 89.4% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T05:31:56.585 INFO:teuthology.orchestra.run.smithi063.stderr: -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T05:31:56.585 INFO:teuthology.orchestra.run.smithi063.stderr: 81.5% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T05:31:56.586 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-24T05:31:56.586 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-volume.log: 84.2% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T05:31:56.586 INFO:teuthology.orchestra.run.smithi063.stderr: 81.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T05:31:56.586 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-mon.b.log: 90.5% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T05:31:56.587 INFO:teuthology.orchestra.run.smithi143.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T05:31:56.587 INFO:teuthology.orchestra.run.smithi143.stderr:: No such file or directory 2024-03-24T05:31:56.587 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T05:31:56.587 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-mgr.z.log 2024-03-24T05:31:56.587 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T05:31:56.588 INFO:teuthology.orchestra.run.smithi063.stderr: 90.2% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-24T05:31:56.588 INFO:teuthology.orchestra.run.smithi023.stderr: 90.5% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T05:31:56.590 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.audit.log: /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.log 2024-03-24T05:31:56.590 INFO:teuthology.orchestra.run.smithi143.stderr: 90.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T05:31:56.590 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-24T05:31:56.590 INFO:teuthology.orchestra.run.smithi143.stderr: 81.4% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T05:31:56.591 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T05:31:56.591 INFO:teuthology.orchestra.run.smithi143.stderr: 89.5% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T05:31:56.591 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-24T05:31:56.591 INFO:teuthology.orchestra.run.smithi143.stderr: 84.3% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T05:31:56.591 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-mon.c.log: 90.1% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-mgr.z.log.gz 2024-03-24T05:31:56.591 INFO:teuthology.orchestra.run.smithi143.stderr: 90.5% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T05:31:56.633 INFO:teuthology.orchestra.run.smithi063.stderr: 92.2% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-24T05:31:56.638 INFO:teuthology.orchestra.run.smithi143.stderr: 92.2% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-24T05:31:56.646 INFO:teuthology.orchestra.run.smithi023.stderr: 88.9% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-24T05:31:56.755 INFO:teuthology.orchestra.run.smithi023.stderr: 91.3% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-24T05:31:56.862 INFO:teuthology.orchestra.run.smithi143.stderr: 94.2% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-24T05:31:56.864 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-03-24T05:31:56.864 INFO:teuthology.orchestra.run.smithi143.stderr:real 0m0.295s 2024-03-24T05:31:56.864 INFO:teuthology.orchestra.run.smithi143.stderr:user 0m0.346s 2024-03-24T05:31:56.865 INFO:teuthology.orchestra.run.smithi143.stderr:sys 0m0.028s 2024-03-24T05:31:56.867 INFO:teuthology.orchestra.run.smithi023.stderr: 94.3% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-24T05:31:56.869 INFO:teuthology.orchestra.run.smithi023.stderr: 2024-03-24T05:31:56.869 INFO:teuthology.orchestra.run.smithi023.stderr:real 0m0.304s 2024-03-24T05:31:56.869 INFO:teuthology.orchestra.run.smithi023.stderr:user 0m0.536s 2024-03-24T05:31:56.869 INFO:teuthology.orchestra.run.smithi023.stderr:sys 0m0.038s 2024-03-24T05:31:56.914 INFO:teuthology.orchestra.run.smithi063.stderr: 94.2% -- replaced with /var/log/ceph/e2aa55c2-e99e-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-24T05:31:56.916 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-24T05:31:56.916 INFO:teuthology.orchestra.run.smithi063.stderr:real 0m0.350s 2024-03-24T05:31:56.916 INFO:teuthology.orchestra.run.smithi063.stderr:user 0m0.394s 2024-03-24T05:31:56.916 INFO:teuthology.orchestra.run.smithi063.stderr:sys 0m0.038s 2024-03-24T05:31:56.916 INFO:tasks.cephadm:Archiving logs... 2024-03-24T05:31:56.917 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617855/remote/smithi023/log 2024-03-24T05:31:56.918 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T05:31:57.047 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617855/remote/smithi063/log 2024-03-24T05:31:57.048 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T05:31:57.134 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617855/remote/smithi143/log 2024-03-24T05:31:57.135 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T05:31:57.218 INFO:tasks.cephadm:Removing cluster... 2024-03-24T05:31:57.218 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 --force 2024-03-24T05:31:57.402 INFO:teuthology.orchestra.run.smithi023.stdout:Deleting cluster with fsid: e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:31:57.514 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 --force 2024-03-24T05:31:57.699 INFO:teuthology.orchestra.run.smithi063.stdout:Deleting cluster with fsid: e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:31:57.813 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e2aa55c2-e99e-11ee-95cd-87774f69a715 --force 2024-03-24T05:31:57.998 INFO:teuthology.orchestra.run.smithi143.stdout:Deleting cluster with fsid: e2aa55c2-e99e-11ee-95cd-87774f69a715 2024-03-24T05:31:58.118 INFO:tasks.cephadm:Removing cephadm ... 2024-03-24T05:31:58.118 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T05:31:58.135 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T05:31:58.151 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T05:31:58.165 INFO:tasks.cephadm:Teardown complete 2024-03-24T05:31:58.166 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-24T05:31:58.176 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi023:/dev/vg_nvme/lv_1... 2024-03-24T05:31:58.176 DEBUG:teuthology.orchestra.run.smithi023:> sudo nvme disconnect -n lv_1 2024-03-24T05:31:58.317 INFO:teuthology.orchestra.run.smithi023.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-24T05:31:58.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:31:58.319 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi023:/dev/vg_nvme/lv_2... 2024-03-24T05:31:58.319 DEBUG:teuthology.orchestra.run.smithi023:> sudo nvme disconnect -n lv_2 2024-03-24T05:31:58.476 INFO:teuthology.orchestra.run.smithi023.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-24T05:31:58.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:31:58.478 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi023:/dev/vg_nvme/lv_3... 2024-03-24T05:31:58.478 DEBUG:teuthology.orchestra.run.smithi023:> sudo nvme disconnect -n lv_3 2024-03-24T05:31:58.644 INFO:teuthology.orchestra.run.smithi023.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-24T05:31:58.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:31:58.646 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi023:/dev/vg_nvme/lv_4... 2024-03-24T05:31:58.646 DEBUG:teuthology.orchestra.run.smithi023:> sudo nvme disconnect -n lv_4 2024-03-24T05:31:58.793 INFO:teuthology.orchestra.run.smithi023.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-24T05:31:58.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:31:58.796 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T05:31:58.796 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/scratch_devs 2024-03-24T05:31:58.824 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi063:/dev/vg_nvme/lv_1... 2024-03-24T05:31:58.824 DEBUG:teuthology.orchestra.run.smithi063:> sudo nvme disconnect -n lv_1 2024-03-24T05:31:58.975 INFO:teuthology.orchestra.run.smithi063.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-24T05:31:58.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:31:58.978 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi063:/dev/vg_nvme/lv_2... 2024-03-24T05:31:58.978 DEBUG:teuthology.orchestra.run.smithi063:> sudo nvme disconnect -n lv_2 2024-03-24T05:31:59.137 INFO:teuthology.orchestra.run.smithi063.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-24T05:31:59.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:31:59.139 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi063:/dev/vg_nvme/lv_3... 2024-03-24T05:31:59.139 DEBUG:teuthology.orchestra.run.smithi063:> sudo nvme disconnect -n lv_3 2024-03-24T05:31:59.281 INFO:teuthology.orchestra.run.smithi063.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-24T05:31:59.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:31:59.283 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi063:/dev/vg_nvme/lv_4... 2024-03-24T05:31:59.283 DEBUG:teuthology.orchestra.run.smithi063:> sudo nvme disconnect -n lv_4 2024-03-24T05:31:59.420 INFO:teuthology.orchestra.run.smithi063.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-24T05:31:59.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:31:59.422 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-24T05:31:59.423 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/scratch_devs 2024-03-24T05:31:59.453 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi143:/dev/vg_nvme/lv_1... 2024-03-24T05:31:59.453 DEBUG:teuthology.orchestra.run.smithi143:> sudo nvme disconnect -n lv_1 2024-03-24T05:31:59.589 INFO:teuthology.orchestra.run.smithi143.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-24T05:31:59.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:31:59.591 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi143:/dev/vg_nvme/lv_2... 2024-03-24T05:31:59.591 DEBUG:teuthology.orchestra.run.smithi143:> sudo nvme disconnect -n lv_2 2024-03-24T05:31:59.729 INFO:teuthology.orchestra.run.smithi143.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-24T05:31:59.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:31:59.731 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi143:/dev/vg_nvme/lv_3... 2024-03-24T05:31:59.731 DEBUG:teuthology.orchestra.run.smithi143:> sudo nvme disconnect -n lv_3 2024-03-24T05:31:59.889 INFO:teuthology.orchestra.run.smithi143.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-24T05:31:59.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:31:59.891 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi143:/dev/vg_nvme/lv_4... 2024-03-24T05:31:59.891 DEBUG:teuthology.orchestra.run.smithi143:> sudo nvme disconnect -n lv_4 2024-03-24T05:32:00.023 INFO:teuthology.orchestra.run.smithi143.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-24T05:32:00.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:32:00.025 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T05:32:00.025 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/scratch_devs 2024-03-24T05:32:00.053 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T05:32:00.063 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T05:32:00.063 DEBUG:teuthology.orchestra.run.smithi023:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T05:32:00.067 DEBUG:teuthology.orchestra.run.smithi063:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T05:32:00.070 DEBUG:teuthology.orchestra.run.smithi143:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T05:32:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:bash: line 1: ntpq: command not found 2024-03-24T05:32:00.083 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T05:32:00.083 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2024-03-24T05:32:00.083 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 -183us[ -183us] +/- 82ms 2024-03-24T05:32:00.083 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 23 -133us[ -133us] +/- 77ms 2024-03-24T05:32:00.084 INFO:teuthology.orchestra.run.smithi023.stdout:^* hv03.front.sepia.ceph.com 3 6 377 24 +324us[ +332us] +/- 68ms 2024-03-24T05:32:00.084 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T05:32:00.084 INFO:teuthology.orchestra.run.smithi063.stderr:bash: line 1: ntpq: command not found 2024-03-24T05:32:00.088 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T05:32:00.088 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-03-24T05:32:00.088 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 -197us[ -197us] +/- 82ms 2024-03-24T05:32:00.088 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 23 -143us[ -134us] +/- 77ms 2024-03-24T05:32:00.088 INFO:teuthology.orchestra.run.smithi063.stdout:^* hv03.front.sepia.ceph.com 3 6 377 23 +294us[ +303us] +/- 68ms 2024-03-24T05:32:00.088 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T05:32:00.107 INFO:teuthology.orchestra.run.smithi143.stderr:bash: line 1: ntpq: command not found 2024-03-24T05:32:00.112 INFO:teuthology.orchestra.run.smithi143.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T05:32:00.112 INFO:teuthology.orchestra.run.smithi143.stdout:=============================================================================== 2024-03-24T05:32:00.112 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 -184us[ -184us] +/- 82ms 2024-03-24T05:32:00.112 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 22 -134us[ -134us] +/- 77ms 2024-03-24T05:32:00.112 INFO:teuthology.orchestra.run.smithi143.stdout:^* hv03.front.sepia.ceph.com 3 6 377 26 +305us[ +318us] +/- 68ms 2024-03-24T05:32:00.112 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T05:32:00.113 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T05:32:00.124 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T05:32:00.126 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T05:32:00.137 DEBUG:teuthology.orchestra.run.smithi023:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T05:32:00.211 DEBUG:teuthology.orchestra.run.smithi063:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T05:32:00.287 DEBUG:teuthology.orchestra.run.smithi143:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T05:32:00.356 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T05:32:00.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:32:00.386 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T05:32:00.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:32:00.418 DEBUG:teuthology.orchestra.run.smithi143:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T05:32:00.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:32:00.449 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T05:32:00.459 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T05:32:00.471 INFO:teuthology.task.internal:Duration was 668.094057 seconds 2024-03-24T05:32:00.471 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T05:32:00.492 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T05:32:00.492 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T05:32:00.495 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T05:32:00.497 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T05:32:00.535 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T05:32:00.540 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T05:32:00.540 INFO:teuthology.orchestra.run.smithi143.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T05:32:00.865 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-24T05:32:00.865 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi023.front.sepia.ceph.com 2024-03-24T05:32:00.865 DEBUG:teuthology.orchestra.run.smithi023:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T05:32:00.925 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi063.front.sepia.ceph.com 2024-03-24T05:32:00.925 DEBUG:teuthology.orchestra.run.smithi063:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T05:32:00.953 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi143.front.sepia.ceph.com 2024-03-24T05:32:00.953 DEBUG:teuthology.orchestra.run.smithi143:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T05:32:00.976 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-24T05:32:00.977 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T05:32:00.980 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T05:32:00.996 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T05:32:01.045 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-24T05:32:01.045 DEBUG:teuthology.orchestra.run.smithi023:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T05:32:01.048 DEBUG:teuthology.orchestra.run.smithi063:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T05:32:01.066 DEBUG:teuthology.orchestra.run.smithi143:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T05:32:01.236 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T05:32:01.246 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T05:32:01.247 DEBUG:teuthology.orchestra.run.smithi023:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T05:32:01.273 DEBUG:teuthology.orchestra.run.smithi063:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T05:32:01.301 DEBUG:teuthology.orchestra.run.smithi143:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T05:32:01.328 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T05:32:01.338 DEBUG:teuthology.orchestra.run.smithi023:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T05:32:01.341 DEBUG:teuthology.orchestra.run.smithi063:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T05:32:01.344 DEBUG:teuthology.orchestra.run.smithi143:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T05:32:01.364 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = core 2024-03-24T05:32:01.366 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = core 2024-03-24T05:32:01.394 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = core 2024-03-24T05:32:01.447 DEBUG:teuthology.orchestra.run.smithi023:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T05:32:01.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:32:01.462 DEBUG:teuthology.orchestra.run.smithi063:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T05:32:01.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:32:01.476 DEBUG:teuthology.orchestra.run.smithi143:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T05:32:01.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T05:32:01.503 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T05:32:01.513 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T05:32:01.514 DEBUG:teuthology.misc:Transferring archived files from smithi023:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617855/remote/smithi023 2024-03-24T05:32:01.514 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T05:32:01.556 DEBUG:teuthology.misc:Transferring archived files from smithi063:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617855/remote/smithi063 2024-03-24T05:32:01.556 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T05:32:01.595 DEBUG:teuthology.misc:Transferring archived files from smithi143:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617855/remote/smithi143 2024-03-24T05:32:01.595 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T05:32:01.633 INFO:teuthology.task.internal:Removing archive directory... 2024-03-24T05:32:01.633 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T05:32:01.636 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T05:32:01.638 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T05:32:01.682 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T05:32:01.693 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T05:32:01.693 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T05:32:01.708 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T05:32:01.708 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T05:32:01.710 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T05:32:01.713 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T05:32:01.722 INFO:teuthology.orchestra.run.smithi023.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 05:32 /home/ubuntu/cephtest 2024-03-24T05:32:01.726 INFO:teuthology.orchestra.run.smithi063.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 05:32 /home/ubuntu/cephtest 2024-03-24T05:32:01.736 INFO:teuthology.orchestra.run.smithi143.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 05:32 /home/ubuntu/cephtest 2024-03-24T05:32:01.737 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T05:32:01.750 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T05:32:01.777 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T05:32:01.803 DEBUG:teuthology.nuke:shortname: smithi023 2024-03-24T05:32:01.803 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T05:32:01.828 DEBUG:teuthology.nuke:shortname: smithi063 2024-03-24T05:32:01.829 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T05:32:01.834 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617855', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 05:10:06.041341', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMlgx8TU2EZZJlDCNWuEJbkInhWgJsX/RQA0JH2FoA7VmjYF5vqLjUE3+rKq8eQXwWyvYa5UrI/dPohW+ASaLbo='} 2024-03-24T05:32:01.853 DEBUG:teuthology.nuke:shortname: smithi143 2024-03-24T05:32:01.854 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T05:32:01.858 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617855', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 05:10:06.039004', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPXnArZi4/aoVTQ0ccXCY8lD2PjU4Vhg/cgJLyN9uzbSqNRcpbsiWRuuJxr1JMN05J/Jwnp9V3VXmIHIwynPscU='} 2024-03-24T05:32:01.887 INFO:teuthology.orchestra.console.smithi023:Power off 2024-03-24T05:32:01.887 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T05:32:01.898 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617855', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 05:10:06.043398', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKuztj9tdgI3H2Syq6QpP/oB/a+drUzJLKyUHlHCqTbap1yDLzVWSathX1KkSA85hlMXQ3upw1xLMrjI0a4CucE='} 2024-03-24T05:32:01.915 DEBUG:teuthology.orchestra.console.smithi023:power off output: Chassis Power Control: Down/Off 2024-03-24T05:32:01.915 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T05:32:01.933 INFO:teuthology.orchestra.console.smithi063:Power off 2024-03-24T05:32:01.933 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T05:32:01.947 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2024-03-24T05:32:01.955 DEBUG:teuthology.orchestra.console.smithi063:power off output: Chassis Power Control: Down/Off 2024-03-24T05:32:01.955 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T05:32:01.974 INFO:teuthology.orchestra.console.smithi143:Power off 2024-03-24T05:32:01.975 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T05:32:01.984 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is on 2024-03-24T05:32:01.998 DEBUG:teuthology.orchestra.console.smithi143:power off output: Chassis Power Control: Down/Off 2024-03-24T05:32:01.998 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T05:32:02.023 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on 2024-03-24T05:32:05.948 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T05:32:05.985 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T05:32:06.024 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T05:32:06.059 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2024-03-24T05:32:06.096 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is on 2024-03-24T05:32:06.134 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on 2024-03-24T05:32:10.065 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T05:32:10.097 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T05:32:10.135 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T05:32:10.178 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2024-03-24T05:32:10.208 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is on 2024-03-24T05:32:10.245 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on 2024-03-24T05:32:14.184 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T05:32:14.209 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T05:32:14.246 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T05:32:14.297 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is off 2024-03-24T05:32:14.318 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is off 2024-03-24T05:32:14.355 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is off 2024-03-24T05:32:14.398 INFO:teuthology.orchestra.console.smithi023:Power off completed 2024-03-24T05:32:14.419 INFO:teuthology.orchestra.console.smithi063:Power off completed 2024-03-24T05:32:14.456 INFO:teuthology.orchestra.console.smithi143:Power off completed 2024-03-24T05:32:14.560 INFO:teuthology.run:Summary data: description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} duration: 668.0940570831299 failure_reason: '"2024-03-24T05:28:10.713557+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-24T05:32:14.560 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T05:32:14.652 INFO:teuthology.run:FAIL