2024-03-11T22:03:48.599 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-11T22:03:48.599 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591967 --owner scheduled_teuthology@teuthology --name teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi --verbose --description orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} -- /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591967/orig.config.yaml 2024-03-11T22:03:48.626 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T22:03:48.703 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591967 branch: squid description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591967' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 94a539707733325322964c6a187c6a1b41be1daf ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 94a539707733325322964c6a187c6a1b41be1daf selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: fbcdfe674e56d1ea7a0810744e5915bb07961be5 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.1 - client.1 - prometheus.a - grafana.a - node-exporter.b - - mon.c - mgr.z - osd.2 - client.2 - node-exporter.c seed: 3176 sha1: 94a539707733325322964c6a187c6a1b41be1daf sleep_before_teardown: 0 subset: 14/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: fbcdfe674e56d1ea7a0810744e5915bb07961be5 targets: smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA4R4hiLP6CWK8D+CcttU1Rmgt5UhEor2Zo2LA3JP1z2cTomjDnn8/EjFiVnSVuVQ7PCXs7yOnMCyEfTsN/l7tc= smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI3ptjM9LOuwQkz+C+zsIjrpORomAcZIktI+ec1eN/X4K3amWAUhGhL2onb3vwNvwQyVKogLHjx1bXYOCdQHEjM= smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ9MaOw7ifKmwoCA9PXLsbUa415InpwWbpwXyvjlVtLGEJmbsS5yh5XZqxpJpCO/1HWyTdjpnD6sfTQn7yDgN6A= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-11T22:03:48.703 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa; will attempt to use it 2024-03-11T22:03:48.704 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks 2024-03-11T22:03:48.704 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-11T22:03:48.706 INFO:teuthology.task.internal:Checking packages... 2024-03-11T22:03:48.729 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '94a539707733325322964c6a187c6a1b41be1daf' 2024-03-11T22:03:48.729 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T22:03:48.730 INFO:teuthology.packaging:ref: None 2024-03-11T22:03:48.730 INFO:teuthology.packaging:tag: None 2024-03-11T22:03:48.730 INFO:teuthology.packaging:branch: squid 2024-03-11T22:03:48.730 INFO:teuthology.packaging:sha1: 94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:03:48.730 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-11T22:03:49.087 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1607.g94a53970 2024-03-11T22:03:49.088 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-11T22:03:49.098 INFO:teuthology.task.internal:no buildpackages task found 2024-03-11T22:03:49.098 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-11T22:03:49.118 INFO:teuthology.task.internal:Saving configuration 2024-03-11T22:03:49.128 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-11T22:03:49.146 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:03:49.170 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591967', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:57:56.790260', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA4R4hiLP6CWK8D+CcttU1Rmgt5UhEor2Zo2LA3JP1z2cTomjDnn8/EjFiVnSVuVQ7PCXs7yOnMCyEfTsN/l7tc='} 2024-03-11T22:03:49.195 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591967', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:57:56.792695', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI3ptjM9LOuwQkz+C+zsIjrpORomAcZIktI+ec1eN/X4K3amWAUhGhL2onb3vwNvwQyVKogLHjx1bXYOCdQHEjM='} 2024-03-11T22:03:49.217 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591967', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:57:56.791882', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ9MaOw7ifKmwoCA9PXLsbUa415InpwWbpwXyvjlVtLGEJmbsS5yh5XZqxpJpCO/1HWyTdjpnD6sfTQn7yDgN6A='} 2024-03-11T22:03:49.217 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-11T22:03:49.225 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-11T22:03:49.225 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.1', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-11T22:03:49.225 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'osd.2', 'client.2', 'node-exporter.c'] 2024-03-11T22:03:49.225 INFO:teuthology.run_tasks:Running task console_log... 2024-03-11T22:03:49.342 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f35a7f79f70>, signals=[15]) 2024-03-11T22:03:49.343 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-11T22:03:49.348 INFO:teuthology.task.internal:Opening connections... 2024-03-11T22:03:49.348 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2024-03-11T22:03:49.350 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:03:49.454 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi174.front.sepia.ceph.com 2024-03-11T22:03:49.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:03:49.541 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2024-03-11T22:03:49.542 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:03:49.630 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-11T22:03:49.636 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2024-03-11T22:03:49.654 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2024-03-11T22:03:49.654 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2024-03-11T22:03:49.714 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Stream" 2024-03-11T22:03:49.715 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="9" 2024-03-11T22:03:49.715 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos" 2024-03-11T22:03:49.715 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora" 2024-03-11T22:03:49.715 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="9" 2024-03-11T22:03:49.715 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el9" 2024-03-11T22:03:49.715 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T22:03:49.715 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2024-03-11T22:03:49.715 INFO:teuthology.orchestra.run.smithi027.stdout:LOGO="fedora-logo-icon" 2024-03-11T22:03:49.715 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T22:03:49.715 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/" 2024-03-11T22:03:49.715 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T22:03:49.715 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T22:03:49.715 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T22:03:49.716 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2024-03-11T22:03:49.741 DEBUG:teuthology.orchestra.run.smithi174:> uname -m 2024-03-11T22:03:49.761 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2024-03-11T22:03:49.761 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release 2024-03-11T22:03:49.818 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="CentOS Stream" 2024-03-11T22:03:49.818 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="9" 2024-03-11T22:03:49.818 INFO:teuthology.orchestra.run.smithi174.stdout:ID="centos" 2024-03-11T22:03:49.818 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE="rhel fedora" 2024-03-11T22:03:49.818 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="9" 2024-03-11T22:03:49.818 INFO:teuthology.orchestra.run.smithi174.stdout:PLATFORM_ID="platform:el9" 2024-03-11T22:03:49.818 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T22:03:49.818 INFO:teuthology.orchestra.run.smithi174.stdout:ANSI_COLOR="0;31" 2024-03-11T22:03:49.819 INFO:teuthology.orchestra.run.smithi174.stdout:LOGO="fedora-logo-icon" 2024-03-11T22:03:49.819 INFO:teuthology.orchestra.run.smithi174.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T22:03:49.819 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://centos.org/" 2024-03-11T22:03:49.819 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T22:03:49.819 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T22:03:49.819 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T22:03:49.819 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2024-03-11T22:03:49.841 DEBUG:teuthology.orchestra.run.smithi183:> uname -m 2024-03-11T22:03:49.859 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2024-03-11T22:03:49.859 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2024-03-11T22:03:49.917 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="CentOS Stream" 2024-03-11T22:03:49.917 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="9" 2024-03-11T22:03:49.917 INFO:teuthology.orchestra.run.smithi183.stdout:ID="centos" 2024-03-11T22:03:49.917 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="rhel fedora" 2024-03-11T22:03:49.917 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="9" 2024-03-11T22:03:49.917 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el9" 2024-03-11T22:03:49.917 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T22:03:49.917 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2024-03-11T22:03:49.917 INFO:teuthology.orchestra.run.smithi183.stdout:LOGO="fedora-logo-icon" 2024-03-11T22:03:49.917 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T22:03:49.918 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://centos.org/" 2024-03-11T22:03:49.918 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T22:03:49.918 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T22:03:49.918 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T22:03:49.918 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2024-03-11T22:03:49.941 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-11T22:03:49.953 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-11T22:03:49.970 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-11T22:03:49.970 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2024-03-11T22:03:49.972 DEBUG:teuthology.orchestra.run.smithi174:> test '!' -e /home/ubuntu/cephtest 2024-03-11T22:03:49.975 DEBUG:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2024-03-11T22:03:49.991 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-11T22:03:49.997 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-11T22:03:49.997 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2024-03-11T22:03:50.030 DEBUG:teuthology.orchestra.run.smithi174:> test -z $(ls -A /var/lib/ceph) 2024-03-11T22:03:50.032 DEBUG:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2024-03-11T22:03:50.147 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-11T22:03:50.226 INFO:teuthology.run_tasks:Running task kernel... 2024-03-11T22:03:50.240 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-11T22:03:50.240 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-11T22:03:50.240 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T22:03:50.241 DEBUG:teuthology.orchestra.run.smithi174:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T22:03:50.241 DEBUG:teuthology.orchestra.run.smithi183:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T22:03:50.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:03:50.257 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2024-03-11T22:03:50.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:03:50.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:03:50.259 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-03-11T22:03:50.259 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-11T22:03:50.314 INFO:teuthology.orchestra.run.smithi174.stdout:5.14.0-361.el9.x86_64 2024-03-11T22:03:50.314 INFO:teuthology.task.kernel:Running kernel on smithi174: 5.14.0-361.el9.x86_64 2024-03-11T22:03:50.314 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum install -y kernel 2024-03-11T22:03:50.316 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-361.el9.x86_64 2024-03-11T22:03:50.316 INFO:teuthology.task.kernel:Running kernel on smithi183: 5.14.0-361.el9.x86_64 2024-03-11T22:03:50.316 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum install -y kernel 2024-03-11T22:03:50.317 INFO:teuthology.orchestra.run.smithi027.stdout:5.14.0-361.el9.x86_64 2024-03-11T22:03:50.318 INFO:teuthology.task.kernel:Running kernel on smithi027: 5.14.0-361.el9.x86_64 2024-03-11T22:03:50.318 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum install -y kernel 2024-03-11T22:03:52.530 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-11T22:03:52.530 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-11T22:03:52.535 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-11T22:03:54.086 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-9 - BaseOS 4.9 MB/s | 2.0 MB 00:00 2024-03-11T22:03:54.098 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS-9 - BaseOS 4.3 MB/s | 2.0 MB 00:00 2024-03-11T22:03:54.313 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 2.9 MB/s | 2.0 MB 00:00 2024-03-11T22:03:56.143 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-03-11T22:03:56.607 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - BaseOS 3.7 MB/s | 8.0 MB 00:02 2024-03-11T22:03:56.837 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 3.7 MB/s | 8.0 MB 00:02 2024-03-11T22:03:58.555 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-11T22:04:06.651 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 2.0 MB/s | 19 MB 00:09 2024-03-11T22:04:21.945 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - CRB 315 kB/s | 6.1 MB 00:19 2024-03-11T22:04:23.329 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 15 kB 00:00 2024-03-11T22:04:25.025 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-11T22:04:29.939 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-11T22:04:32.138 INFO:teuthology.orchestra.run.smithi174.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T22:04:32.138 INFO:teuthology.orchestra.run.smithi174.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T22:04:32.235 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-11T22:04:32.241 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-11T22:04:32.241 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2024-03-11T22:04:32.241 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-11T22:04:32.241 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-11T22:04:32.241 INFO:teuthology.orchestra.run.smithi174.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T22:04:32.241 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-03-11T22:04:32.241 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T22:04:32.241 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T22:04:32.241 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T22:04:32.241 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-11T22:04:32.241 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-11T22:04:32.241 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-11T22:04:32.242 INFO:teuthology.orchestra.run.smithi174.stdout:Install 4 Packages 2024-03-11T22:04:32.242 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-11T22:04:32.242 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 100 M 2024-03-11T22:04:32.242 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 123 M 2024-03-11T22:04:32.242 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-11T22:04:39.883 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 212 kB/s | 6.1 MB 00:29 2024-03-11T22:04:42.314 INFO:teuthology.orchestra.run.smithi174.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 644 kB/s | 6.0 MB 00:09 2024-03-11T22:04:42.525 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 15 kB 00:01 2024-03-11T22:04:44.312 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-11T22:04:47.830 INFO:teuthology.orchestra.run.smithi174.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 1.4 MB/s | 21 MB 00:15 2024-03-11T22:04:49.485 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-03-11T22:04:50.372 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 368 kB/s | 19 MB 00:52 2024-03-11T22:04:51.792 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T22:04:51.792 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T22:04:51.883 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-11T22:04:51.888 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-11T22:04:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-03-11T22:04:51.888 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-11T22:04:51.888 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-11T22:04:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T22:04:51.888 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-03-11T22:04:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T22:04:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T22:04:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T22:04:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:04:51.889 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-11T22:04:51.889 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-11T22:04:51.889 INFO:teuthology.orchestra.run.smithi027.stdout:Install 4 Packages 2024-03-11T22:04:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:04:51.889 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 100 M 2024-03-11T22:04:51.890 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 123 M 2024-03-11T22:04:51.890 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-11T22:04:53.195 INFO:teuthology.orchestra.run.smithi027.stdout:(1/4): kernel-core-5.14.0-427.el9.x86_64.rpm 17 MB/s | 21 MB 00:01 2024-03-11T22:04:53.428 INFO:teuthology.orchestra.run.smithi027.stdout:(2/4): kernel-5.14.0-427.el9.x86_64.rpm 4.0 MB/s | 6.0 MB 00:01 2024-03-11T22:04:54.478 INFO:teuthology.orchestra.run.smithi027.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 26 MB/s | 33 MB 00:01 2024-03-11T22:04:56.203 INFO:teuthology.orchestra.run.smithi027.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 9.2 MB/s | 39 MB 00:04 2024-03-11T22:04:56.206 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:04:56.207 INFO:teuthology.orchestra.run.smithi027.stdout:Total 23 MB/s | 100 MB 00:04 2024-03-11T22:04:57.125 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-11T22:04:57.571 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-11T22:04:57.571 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-11T22:04:57.672 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.1 MB 00:03 2024-03-11T22:04:57.763 INFO:teuthology.orchestra.run.smithi174.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 2.2 MB/s | 33 MB 00:15 2024-03-11T22:04:58.113 INFO:teuthology.orchestra.run.smithi174.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 1.6 MB/s | 39 MB 00:25 2024-03-11T22:04:58.113 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:04:58.113 INFO:teuthology.orchestra.run.smithi174.stdout:Total 3.8 MB/s | 100 MB 00:25 2024-03-11T22:04:58.990 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-11T22:04:58.990 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-11T22:04:59.001 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-11T22:04:59.398 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-11T22:04:59.398 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-11T22:05:00.339 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 10 kB/s | 15 kB 00:01 2024-03-11T22:05:01.016 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-11T22:05:01.016 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-11T22:05:01.853 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-11T22:05:02.526 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:05:02.662 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:05:03.781 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-11T22:05:04.436 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:05:04.564 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:05:05.592 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:05:05.766 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:05:07.360 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:05:07.540 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:05:10.866 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:05:11.171 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:05:12.320 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:05:12.449 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:05:16.294 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:05:17.211 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:05:46.246 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:05:46.247 INFO:teuthology.orchestra.run.smithi174.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T22:05:46.247 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-11T22:05:46.331 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:05:46.819 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:05:46.819 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:05:46.819 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:05:46.820 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:05:46.940 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:05:46.940 INFO:teuthology.orchestra.run.smithi027.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T22:05:46.940 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:05:47.028 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:05:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:05:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-11T22:05:47.271 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-11T22:05:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T22:05:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T22:05:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T22:05:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T22:05:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-11T22:05:47.271 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-11T22:05:47.435 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:05:47.436 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:05:47.436 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:05:47.436 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:05:47.751 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:05:47.857 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:05:47.857 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:05:47.857 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-11T22:05:47.857 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T22:05:47.857 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T22:05:47.857 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T22:05:47.857 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T22:05:47.857 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:05:47.857 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-11T22:05:47.889 INFO:teuthology.orchestra.run.smithi174.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:05:47.889 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:05:47.889 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T22:05:47.889 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T22:05:47.889 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T22:05:47.889 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-11T22:05:47.890 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi174.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T22:05:47.890 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:05:48.010 INFO:teuthology.orchestra.run.smithi174.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:05:48.010 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:05:48.010 DEBUG:teuthology.orchestra.run.smithi174:> sudo rpm -qi grub2-tools 2024-03-11T22:05:48.044 INFO:teuthology.orchestra.run.smithi174.stdout:Name : grub2-tools 2024-03-11T22:05:48.044 INFO:teuthology.orchestra.run.smithi174.stdout:Epoch : 1 2024-03-11T22:05:48.044 INFO:teuthology.orchestra.run.smithi174.stdout:Version : 2.06 2024-03-11T22:05:48.044 INFO:teuthology.orchestra.run.smithi174.stdout:Release : 68.el9 2024-03-11T22:05:48.044 INFO:teuthology.orchestra.run.smithi174.stdout:Architecture: x86_64 2024-03-11T22:05:48.044 INFO:teuthology.orchestra.run.smithi174.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T22:05:48.045 INFO:teuthology.orchestra.run.smithi174.stdout:Group : Unspecified 2024-03-11T22:05:48.045 INFO:teuthology.orchestra.run.smithi174.stdout:Size : 8264254 2024-03-11T22:05:48.045 INFO:teuthology.orchestra.run.smithi174.stdout:License : GPLv3+ 2024-03-11T22:05:48.045 INFO:teuthology.orchestra.run.smithi174.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T22:05:48.045 INFO:teuthology.orchestra.run.smithi174.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T22:05:48.045 INFO:teuthology.orchestra.run.smithi174.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T22:05:48.045 INFO:teuthology.orchestra.run.smithi174.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T22:05:48.045 INFO:teuthology.orchestra.run.smithi174.stdout:Packager : builder@centos.org 2024-03-11T22:05:48.045 INFO:teuthology.orchestra.run.smithi174.stdout:Vendor : CentOS 2024-03-11T22:05:48.045 INFO:teuthology.orchestra.run.smithi174.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T22:05:48.045 INFO:teuthology.orchestra.run.smithi174.stdout:Summary : Support tools for GRUB. 2024-03-11T22:05:48.045 INFO:teuthology.orchestra.run.smithi174.stdout:Description : 2024-03-11T22:05:48.045 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-11T22:05:48.045 INFO:teuthology.orchestra.run.smithi174.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T22:05:48.045 INFO:teuthology.orchestra.run.smithi174.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T22:05:48.046 INFO:teuthology.orchestra.run.smithi174.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T22:05:48.046 INFO:teuthology.orchestra.run.smithi174.stdout:hardware devices. 2024-03-11T22:05:48.046 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-11T22:05:48.046 INFO:teuthology.orchestra.run.smithi174.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T22:05:48.047 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T22:05:48.047 INFO:teuthology.task.kernel:Updating grub on smithi174 to boot 5.14.0-427.el9.x86_64 2024-03-11T22:05:48.047 DEBUG:teuthology.orchestra.run.smithi174:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T22:05:48.381 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:05:48.589 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:05:48.589 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:05:48.590 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T22:05:48.590 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T22:05:48.590 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T22:05:48.590 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-11T22:05:48.590 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi027.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T22:05:48.590 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:05:48.625 INFO:teuthology.orchestra.run.smithi174.stderr:Generating grub configuration file ... 2024-03-11T22:05:48.717 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:05:48.717 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:05:48.717 DEBUG:teuthology.orchestra.run.smithi027:> sudo rpm -qi grub2-tools 2024-03-11T22:05:48.760 INFO:teuthology.orchestra.run.smithi027.stdout:Name : grub2-tools 2024-03-11T22:05:48.760 INFO:teuthology.orchestra.run.smithi027.stdout:Epoch : 1 2024-03-11T22:05:48.760 INFO:teuthology.orchestra.run.smithi027.stdout:Version : 2.06 2024-03-11T22:05:48.760 INFO:teuthology.orchestra.run.smithi027.stdout:Release : 68.el9 2024-03-11T22:05:48.760 INFO:teuthology.orchestra.run.smithi027.stdout:Architecture: x86_64 2024-03-11T22:05:48.761 INFO:teuthology.orchestra.run.smithi027.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T22:05:48.761 INFO:teuthology.orchestra.run.smithi027.stdout:Group : Unspecified 2024-03-11T22:05:48.761 INFO:teuthology.orchestra.run.smithi027.stdout:Size : 8264254 2024-03-11T22:05:48.761 INFO:teuthology.orchestra.run.smithi027.stdout:License : GPLv3+ 2024-03-11T22:05:48.761 INFO:teuthology.orchestra.run.smithi027.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T22:05:48.761 INFO:teuthology.orchestra.run.smithi027.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T22:05:48.761 INFO:teuthology.orchestra.run.smithi027.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T22:05:48.761 INFO:teuthology.orchestra.run.smithi027.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T22:05:48.761 INFO:teuthology.orchestra.run.smithi027.stdout:Packager : builder@centos.org 2024-03-11T22:05:48.761 INFO:teuthology.orchestra.run.smithi027.stdout:Vendor : CentOS 2024-03-11T22:05:48.761 INFO:teuthology.orchestra.run.smithi027.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T22:05:48.761 INFO:teuthology.orchestra.run.smithi027.stdout:Summary : Support tools for GRUB. 2024-03-11T22:05:48.761 INFO:teuthology.orchestra.run.smithi027.stdout:Description : 2024-03-11T22:05:48.761 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:05:48.761 INFO:teuthology.orchestra.run.smithi027.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T22:05:48.761 INFO:teuthology.orchestra.run.smithi027.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T22:05:48.762 INFO:teuthology.orchestra.run.smithi027.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T22:05:48.762 INFO:teuthology.orchestra.run.smithi027.stdout:hardware devices. 2024-03-11T22:05:48.762 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:05:48.762 INFO:teuthology.orchestra.run.smithi027.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T22:05:48.762 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T22:05:48.763 INFO:teuthology.task.kernel:Updating grub on smithi027 to boot 5.14.0-427.el9.x86_64 2024-03-11T22:05:48.763 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T22:05:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:Generating grub configuration file ... 2024-03-11T22:05:50.012 INFO:teuthology.orchestra.run.smithi174.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T22:05:50.044 INFO:teuthology.orchestra.run.smithi174.stderr:done 2024-03-11T22:05:50.050 DEBUG:teuthology.orchestra.run.smithi174:> mktemp 2024-03-11T22:05:50.065 INFO:teuthology.orchestra.run.smithi174.stdout:/tmp/tmp.Wp6vmYMiLi 2024-03-11T22:05:50.065 DEBUG:teuthology.orchestra.run.smithi174:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Wp6vmYMiLi 2024-03-11T22:05:50.134 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0666 /tmp/tmp.Wp6vmYMiLi 2024-03-11T22:05:50.266 DEBUG:teuthology.orchestra.remote:smithi174:/tmp/tmp.Wp6vmYMiLi is 6KB 2024-03-11T22:05:50.286 DEBUG:teuthology.orchestra.run.smithi174:> rm -fr /tmp/tmp.Wp6vmYMiLi 2024-03-11T22:05:50.299 DEBUG:teuthology.orchestra.run.smithi174:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T22:05:50.364 INFO:teuthology.orchestra.run.smithi174.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T22:05:50.364 INFO:teuthology.orchestra.run.smithi174.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T22:05:50.364 INFO:teuthology.orchestra.run.smithi174.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T22:05:50.364 INFO:teuthology.orchestra.run.smithi174.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T22:05:50.365 DEBUG:teuthology.orchestra.run.smithi174:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T22:05:50.502 DEBUG:teuthology.orchestra.run.smithi174:> sudo shutdown -r now 2024-03-11T22:05:50.772 INFO:teuthology.orchestra.run.smithi027.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T22:05:50.785 INFO:teuthology.orchestra.run.smithi027.stderr:done 2024-03-11T22:05:50.791 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2024-03-11T22:05:50.806 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.eJluDFG3QV 2024-03-11T22:05:50.807 DEBUG:teuthology.orchestra.run.smithi027:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.eJluDFG3QV 2024-03-11T22:05:50.879 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.eJluDFG3QV 2024-03-11T22:05:51.031 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.eJluDFG3QV is 6KB 2024-03-11T22:05:51.078 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.eJluDFG3QV 2024-03-11T22:05:51.093 DEBUG:teuthology.orchestra.run.smithi027:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T22:05:51.166 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T22:05:51.166 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T22:05:51.166 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T22:05:51.166 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T22:05:51.168 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T22:05:51.317 DEBUG:teuthology.orchestra.run.smithi027:> sudo shutdown -r now 2024-03-11T22:06:20.504 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-11T22:06:20.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-11T22:06:20.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:06:21.319 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-11T22:06:21.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-11T22:06:21.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:06:38.878 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-03-11T22:06:39.646 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-11T22:06:47.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-11T22:06:47.880 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:06:48.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-11T22:06:48.648 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:06:51.710 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-11T22:07:03.613 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 171 kB/s | 21 MB 02:03 2024-03-11T22:07:03.711 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-11T22:07:03.712 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:07:06.782 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-11T22:07:08.571 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-11T22:07:10.769 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T22:07:10.769 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T22:07:10.908 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-11T22:07:10.918 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-11T22:07:10.918 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-03-11T22:07:10.919 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-11T22:07:10.919 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-11T22:07:10.919 INFO:teuthology.orchestra.run.smithi183.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T22:07:10.919 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-11T22:07:10.919 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T22:07:10.919 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T22:07:10.919 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T22:07:10.919 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-11T22:07:10.919 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-11T22:07:10.919 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-11T22:07:10.919 INFO:teuthology.orchestra.run.smithi183.stdout:Install 4 Packages 2024-03-11T22:07:10.919 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-11T22:07:10.920 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 100 M 2024-03-11T22:07:10.920 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 123 M 2024-03-11T22:07:10.920 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-11T22:07:13.439 INFO:teuthology.orchestra.run.smithi183.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 2.5 MB/s | 6.0 MB 00:02 2024-03-11T22:07:21.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-11T22:07:21.784 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:07:25.346 INFO:teuthology.orchestra.run.smithi183.stdout:(2/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 2.8 MB/s | 39 MB 00:14 2024-03-11T22:07:26.187 INFO:teuthology.orchestra.run.smithi183.stdout:(3/4): kernel-core-5.14.0-427.el9.x86_64.rpm 1.4 MB/s | 21 MB 00:15 2024-03-11T22:07:33.678 INFO:teuthology.orchestra.run.smithi183.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 1.7 MB/s | 33 MB 00:20 2024-03-11T22:07:33.679 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:07:33.679 INFO:teuthology.orchestra.run.smithi183.stdout:Total 4.4 MB/s | 100 MB 00:22 2024-03-11T22:07:34.554 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-11T22:07:34.918 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-11T22:07:34.918 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-11T22:07:36.281 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-11T22:07:36.281 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-11T22:07:39.012 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-11T22:07:39.654 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:07:39.805 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:07:42.604 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:07:42.797 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:07:47.545 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:07:47.732 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:07:47.882 DEBUG:teuthology.orchestra.remote:timed out 2024-03-11T22:07:52.535 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:07:59.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-11T22:07:59.884 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:08:00.210 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-03-11T22:08:00.600 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-11T22:08:00.600 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-11T22:08:00.601 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2024-03-11T22:08:00.616 INFO:teuthology.orchestra.run.smithi174.stdout:5.14.0-427.el9.x86_64 2024-03-11T22:08:00.617 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T22:08:00.617 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T22:08:00.617 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T22:08:01.618 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-11T22:08:01.618 DEBUG:teuthology.orchestra.run.smithi174:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T22:08:01.699 INFO:teuthology.orchestra.run.smithi174.stdout:ttyS1 2024-03-11T22:08:01.732 DEBUG:teuthology.parallel:result is None 2024-03-11T22:08:21.521 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:08:21.521 INFO:teuthology.orchestra.run.smithi183.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T22:08:21.521 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-11T22:08:21.604 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:08:21.787 DEBUG:teuthology.orchestra.remote:timed out 2024-03-11T22:08:22.040 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:08:22.040 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:08:22.040 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:08:22.040 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:08:22.444 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:08:22.444 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-11T22:08:22.444 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-11T22:08:22.444 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T22:08:22.445 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T22:08:22.445 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T22:08:22.445 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T22:08:22.445 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-11T22:08:22.445 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-11T22:08:22.982 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:08:23.130 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:08:23.130 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:08:23.130 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T22:08:23.130 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T22:08:23.130 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T22:08:23.130 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-11T22:08:23.131 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi183.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T22:08:23.131 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:08:23.251 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:08:23.251 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:08:23.251 DEBUG:teuthology.orchestra.run.smithi183:> sudo rpm -qi grub2-tools 2024-03-11T22:08:23.285 INFO:teuthology.orchestra.run.smithi183.stdout:Name : grub2-tools 2024-03-11T22:08:23.285 INFO:teuthology.orchestra.run.smithi183.stdout:Epoch : 1 2024-03-11T22:08:23.285 INFO:teuthology.orchestra.run.smithi183.stdout:Version : 2.06 2024-03-11T22:08:23.285 INFO:teuthology.orchestra.run.smithi183.stdout:Release : 68.el9 2024-03-11T22:08:23.286 INFO:teuthology.orchestra.run.smithi183.stdout:Architecture: x86_64 2024-03-11T22:08:23.286 INFO:teuthology.orchestra.run.smithi183.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T22:08:23.286 INFO:teuthology.orchestra.run.smithi183.stdout:Group : Unspecified 2024-03-11T22:08:23.286 INFO:teuthology.orchestra.run.smithi183.stdout:Size : 8264254 2024-03-11T22:08:23.286 INFO:teuthology.orchestra.run.smithi183.stdout:License : GPLv3+ 2024-03-11T22:08:23.286 INFO:teuthology.orchestra.run.smithi183.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T22:08:23.286 INFO:teuthology.orchestra.run.smithi183.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T22:08:23.286 INFO:teuthology.orchestra.run.smithi183.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T22:08:23.286 INFO:teuthology.orchestra.run.smithi183.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T22:08:23.286 INFO:teuthology.orchestra.run.smithi183.stdout:Packager : builder@centos.org 2024-03-11T22:08:23.286 INFO:teuthology.orchestra.run.smithi183.stdout:Vendor : CentOS 2024-03-11T22:08:23.286 INFO:teuthology.orchestra.run.smithi183.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T22:08:23.287 INFO:teuthology.orchestra.run.smithi183.stdout:Summary : Support tools for GRUB. 2024-03-11T22:08:23.287 INFO:teuthology.orchestra.run.smithi183.stdout:Description : 2024-03-11T22:08:23.287 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-11T22:08:23.287 INFO:teuthology.orchestra.run.smithi183.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T22:08:23.287 INFO:teuthology.orchestra.run.smithi183.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T22:08:23.287 INFO:teuthology.orchestra.run.smithi183.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T22:08:23.287 INFO:teuthology.orchestra.run.smithi183.stdout:hardware devices. 2024-03-11T22:08:23.287 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-11T22:08:23.287 INFO:teuthology.orchestra.run.smithi183.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T22:08:23.288 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T22:08:23.288 INFO:teuthology.task.kernel:Updating grub on smithi183 to boot 5.14.0-427.el9.x86_64 2024-03-11T22:08:23.288 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T22:08:23.895 INFO:teuthology.orchestra.run.smithi183.stderr:Generating grub configuration file ... 2024-03-11T22:08:25.288 INFO:teuthology.orchestra.run.smithi183.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T22:08:25.322 INFO:teuthology.orchestra.run.smithi183.stderr:done 2024-03-11T22:08:25.324 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-03-11T22:08:25.339 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.naTc2hucCp 2024-03-11T22:08:25.339 DEBUG:teuthology.orchestra.run.smithi183:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.naTc2hucCp 2024-03-11T22:08:25.407 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.naTc2hucCp 2024-03-11T22:08:25.557 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.naTc2hucCp is 6KB 2024-03-11T22:08:25.605 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.naTc2hucCp 2024-03-11T22:08:25.619 DEBUG:teuthology.orchestra.run.smithi183:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T22:08:25.686 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T22:08:25.686 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T22:08:25.686 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T22:08:25.686 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T22:08:25.687 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T22:08:25.851 DEBUG:teuthology.orchestra.run.smithi183:> sudo shutdown -r now 2024-03-11T22:08:39.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-11T22:08:39.788 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:08:40.104 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-03-11T22:08:40.483 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-11T22:08:40.483 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-11T22:08:40.483 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-03-11T22:08:40.499 INFO:teuthology.orchestra.run.smithi027.stdout:5.14.0-427.el9.x86_64 2024-03-11T22:08:40.499 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T22:08:40.499 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T22:08:40.499 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T22:08:41.500 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-11T22:08:41.501 DEBUG:teuthology.orchestra.run.smithi027:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T22:08:41.580 INFO:teuthology.orchestra.run.smithi027.stdout:ttyS1 2024-03-11T22:08:41.621 DEBUG:teuthology.parallel:result is None 2024-03-11T22:08:55.854 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-11T22:08:55.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-11T22:08:55.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:09:14.270 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-11T22:09:23.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-11T22:09:23.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:09:26.334 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-11T22:09:38.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-11T22:09:38.336 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:09:56.766 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-11T22:10:11.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-11T22:10:11.768 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:10:12.168 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-11T22:10:12.567 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-11T22:10:12.568 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-427.el9.x86_64"... 2024-03-11T22:10:12.568 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-11T22:10:12.582 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-427.el9.x86_64 2024-03-11T22:10:12.582 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T22:10:12.582 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T22:10:12.582 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T22:10:13.583 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-11T22:10:13.583 DEBUG:teuthology.orchestra.run.smithi183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T22:10:13.663 INFO:teuthology.orchestra.run.smithi183.stdout:ttyS1 2024-03-11T22:10:13.702 DEBUG:teuthology.parallel:result is None 2024-03-11T22:10:13.703 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-11T22:10:13.710 INFO:teuthology.task.internal:Creating test directory... 2024-03-11T22:10:13.710 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T22:10:13.713 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T22:10:13.718 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T22:10:13.732 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-11T22:10:13.738 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-11T22:10:13.765 INFO:teuthology.task.internal:Creating archive directory... 2024-03-11T22:10:13.765 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T22:10:13.772 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T22:10:13.774 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T22:10:13.837 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-11T22:10:13.843 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-11T22:10:13.844 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T22:10:13.846 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T22:10:13.849 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T22:10:13.876 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:10:13.885 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:10:13.888 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:10:13.888 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:10:13.900 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:10:13.906 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:10:13.908 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-11T22:10:13.915 INFO:teuthology.task.internal:Configuring sudo... 2024-03-11T22:10:13.915 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T22:10:13.917 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T22:10:13.945 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T22:10:13.972 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-11T22:10:13.982 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-11T22:10:13.982 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T22:10:13.990 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T22:10:14.015 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T22:10:14.032 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:10:14.087 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:10:14.147 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:10:14.187 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:10:14.234 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-11T22:10:14.235 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T22:10:14.305 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:10:14.349 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:10:14.404 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:10:14.488 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:10:14.533 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-11T22:10:14.533 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T22:10:14.599 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:10:14.653 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:10:14.729 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:10:14.780 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:10:14.846 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-11T22:10:14.846 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T22:10:14.890 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2024-03-11T22:10:14.893 DEBUG:teuthology.orchestra.run.smithi174:> sudo service rsyslog restart 2024-03-11T22:10:14.895 DEBUG:teuthology.orchestra.run.smithi183:> sudo service rsyslog restart 2024-03-11T22:10:14.945 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:10:14.950 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:10:14.997 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:10:15.302 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-11T22:10:15.309 INFO:teuthology.task.internal:Starting timer... 2024-03-11T22:10:15.309 INFO:teuthology.run_tasks:Running task pcp... 2024-03-11T22:10:15.342 INFO:teuthology.run_tasks:Running task selinux... 2024-03-11T22:10:15.373 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-11T22:10:15.374 DEBUG:teuthology.orchestra.run.smithi027:> sudo service auditd rotate 2024-03-11T22:10:15.439 INFO:teuthology.orchestra.run.smithi027.stdout:Rotating logs: 2024-03-11T22:10:15.441 DEBUG:teuthology.orchestra.run.smithi174:> sudo service auditd rotate 2024-03-11T22:10:15.495 INFO:teuthology.orchestra.run.smithi174.stdout:Rotating logs: 2024-03-11T22:10:15.497 DEBUG:teuthology.orchestra.run.smithi183:> sudo service auditd rotate 2024-03-11T22:10:15.571 INFO:teuthology.orchestra.run.smithi183.stdout:Rotating logs: 2024-03-11T22:10:15.573 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-11T22:10:15.573 DEBUG:teuthology.orchestra.run.smithi027:> /usr/sbin/getenforce 2024-03-11T22:10:15.593 INFO:teuthology.orchestra.run.smithi027.stdout:Permissive 2024-03-11T22:10:15.593 DEBUG:teuthology.orchestra.run.smithi174:> /usr/sbin/getenforce 2024-03-11T22:10:15.613 INFO:teuthology.orchestra.run.smithi174.stdout:Permissive 2024-03-11T22:10:15.613 DEBUG:teuthology.orchestra.run.smithi183:> /usr/sbin/getenforce 2024-03-11T22:10:15.682 INFO:teuthology.orchestra.run.smithi183.stdout:Permissive 2024-03-11T22:10:15.683 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi027.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi174.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi183.front.sepia.ceph.com': 'permissive'} 2024-03-11T22:10:15.683 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:10:15.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:15.715 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:10:15.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:15.743 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:10:15.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:15.771 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-11T22:10:15.771 DEBUG:teuthology.orchestra.run.smithi027:> sudo /usr/sbin/setenforce permissive 2024-03-11T22:10:15.812 DEBUG:teuthology.orchestra.run.smithi174:> sudo /usr/sbin/setenforce permissive 2024-03-11T22:10:15.840 DEBUG:teuthology.orchestra.run.smithi183:> sudo /usr/sbin/setenforce permissive 2024-03-11T22:10:16.032 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-11T22:10:16.041 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-11T22:10:16.056 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-11T22:10:16.057 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi027.front.sepia.ceph.com,smithi174.front.sepia.ceph.com,smithi183.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-11T22:14:36.615 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi027.front.sepia.ceph.com'), Remote(name='ubuntu@smithi174.front.sepia.ceph.com'), Remote(name='ubuntu@smithi183.front.sepia.ceph.com')] 2024-03-11T22:14:36.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-11T22:14:36.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:14:36.727 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-03-11T22:14:36.819 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-11T22:14:36.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-11T22:14:36.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:14:36.920 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-03-11T22:14:36.999 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-11T22:14:36.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-11T22:14:37.000 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:14:37.095 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-11T22:14:37.187 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-11T22:14:37.188 INFO:teuthology.run_tasks:Running task clock... 2024-03-11T22:14:37.199 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-11T22:14:37.199 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T22:14:37.200 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:14:37.202 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T22:14:37.203 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:14:37.205 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T22:14:37.205 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:14:37.239 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T22:14:37.240 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T22:14:37.257 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T22:14:37.260 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T22:14:37.262 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T22:14:37.279 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T22:14:37.289 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: ntpd: command not found 2024-03-11T22:14:37.297 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: ntpd: command not found 2024-03-11T22:14:37.303 INFO:teuthology.orchestra.run.smithi174.stdout:506 Cannot talk to daemon 2024-03-11T22:14:37.310 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: ntpd: command not found 2024-03-11T22:14:37.315 INFO:teuthology.orchestra.run.smithi027.stdout:506 Cannot talk to daemon 2024-03-11T22:14:37.321 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T22:14:37.324 INFO:teuthology.orchestra.run.smithi183.stdout:506 Cannot talk to daemon 2024-03-11T22:14:37.335 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T22:14:37.337 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T22:14:37.340 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T22:14:37.355 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T22:14:37.357 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T22:14:37.388 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:14:37.391 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:14:37.391 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-03-11T22:14:37.391 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:14:37.391 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:14:37.391 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:14:37.391 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:14:37.410 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:14:37.411 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:14:37.413 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:14:37.413 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-11T22:14:37.413 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:14:37.413 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:14:37.413 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:14:37.413 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:14:37.415 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:14:37.415 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-03-11T22:14:37.415 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:14:37.415 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:14:37.415 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:14:37.415 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:14:37.416 INFO:teuthology.run_tasks:Running task pexec... 2024-03-11T22:14:37.427 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-11T22:14:37.427 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi027.front.sepia.ceph.com 2024-03-11T22:14:37.428 DEBUG:teuthology.orchestra.run.smithi027:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T22:14:37.428 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi174.front.sepia.ceph.com 2024-03-11T22:14:37.428 DEBUG:teuthology.orchestra.run.smithi174:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T22:14:37.428 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi183.front.sepia.ceph.com 2024-03-11T22:14:37.429 DEBUG:teuthology.orchestra.run.smithi183:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T22:14:38.012 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:57 ago on Mon 11 Mar 2024 10:13:40 PM UTC. 2024-03-11T22:14:38.032 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:56 ago on Mon 11 Mar 2024 10:13:41 PM UTC. 2024-03-11T22:14:38.070 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:49 ago on Mon 11 Mar 2024 10:13:48 PM UTC. 2024-03-11T22:14:38.157 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-11T22:14:38.176 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-11T22:14:38.176 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2024-03-11T22:14:38.176 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-11T22:14:38.176 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-11T22:14:38.176 INFO:teuthology.orchestra.run.smithi174.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T22:14:38.176 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-11T22:14:38.176 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-11T22:14:38.176 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-11T22:14:38.176 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-03-11T22:14:38.176 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-11T22:14:38.177 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-11T22:14:38.179 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-11T22:14:38.179 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-03-11T22:14:38.179 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-11T22:14:38.179 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-11T22:14:38.179 INFO:teuthology.orchestra.run.smithi183.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T22:14:38.179 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-11T22:14:38.179 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-11T22:14:38.179 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-11T22:14:38.179 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-11T22:14:38.179 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-11T22:14:38.180 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 3.1 M 2024-03-11T22:14:38.180 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 9.8 M 2024-03-11T22:14:38.181 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-11T22:14:38.206 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 3.1 M 2024-03-11T22:14:38.206 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 9.8 M 2024-03-11T22:14:38.206 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-11T22:14:38.222 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-11T22:14:38.234 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-11T22:14:38.234 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-03-11T22:14:38.234 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-11T22:14:38.234 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-11T22:14:38.234 INFO:teuthology.orchestra.run.smithi027.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T22:14:38.234 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:14:38.234 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-11T22:14:38.234 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-11T22:14:38.234 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-03-11T22:14:38.235 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:14:38.238 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 3.1 M 2024-03-11T22:14:38.238 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 9.8 M 2024-03-11T22:14:38.239 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-11T22:14:39.195 INFO:teuthology.orchestra.run.smithi174.stdout:runc-1.1.12-2.el9.x86_64.rpm 3.2 MB/s | 3.1 MB 00:00 2024-03-11T22:14:39.195 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:14:39.195 INFO:teuthology.orchestra.run.smithi174.stdout:Total 3.1 MB/s | 3.1 MB 00:01 2024-03-11T22:14:39.232 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-11T22:14:39.239 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-11T22:14:39.240 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-11T22:14:39.319 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-11T22:14:39.319 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-11T22:14:39.407 INFO:teuthology.orchestra.run.smithi183.stdout:runc-1.1.12-2.el9.x86_64.rpm 3.0 MB/s | 3.1 MB 00:01 2024-03-11T22:14:39.408 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:14:39.408 INFO:teuthology.orchestra.run.smithi183.stdout:Total 2.6 MB/s | 3.1 MB 00:01 2024-03-11T22:14:39.445 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-11T22:14:39.451 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-11T22:14:39.451 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-11T22:14:39.530 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-11T22:14:39.531 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-11T22:14:39.532 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-11T22:14:39.614 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:14:39.756 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-11T22:14:39.851 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:14:40.169 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:14:40.440 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:14:40.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:14:40.485 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-11T22:14:40.485 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-11T22:14:40.485 INFO:teuthology.orchestra.run.smithi174.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T22:14:40.485 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-11T22:14:40.485 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-11T22:14:40.726 DEBUG:teuthology.parallel:result is None 2024-03-11T22:14:40.731 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:14:40.731 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-11T22:14:40.731 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-11T22:14:40.731 INFO:teuthology.orchestra.run.smithi183.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T22:14:40.731 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-11T22:14:40.732 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-11T22:14:40.937 DEBUG:teuthology.parallel:result is None 2024-03-11T22:14:43.331 INFO:teuthology.orchestra.run.smithi027.stdout:runc-1.1.12-2.el9.x86_64.rpm 644 kB/s | 3.1 MB 00:04 2024-03-11T22:14:43.332 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:14:43.332 INFO:teuthology.orchestra.run.smithi027.stdout:Total 629 kB/s | 3.1 MB 00:05 2024-03-11T22:14:43.372 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-11T22:14:43.379 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-11T22:14:43.379 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-11T22:14:43.465 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-11T22:14:43.466 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-11T22:14:43.716 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-11T22:14:43.820 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:14:44.439 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:14:44.774 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:14:44.775 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:14:44.775 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-11T22:14:44.775 INFO:teuthology.orchestra.run.smithi027.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T22:14:44.775 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:14:44.775 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-11T22:14:44.998 DEBUG:teuthology.parallel:result is None 2024-03-11T22:14:44.999 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-11T22:14:45.009 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-11T22:14:45.009 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-11T22:14:45.009 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:14:45.024 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T22:14:45.025 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2024-03-11T22:14:45.080 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T22:14:45.080 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:14:45.080 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-11T22:14:45.080 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:14:45.080 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:14:45.080 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-03-11 22:14:35.518262165 +0000 2024-03-11T22:14:45.080 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-11 22:14:13.940790510 +0000 2024-03-11T22:14:45.080 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-11 22:14:13.940790510 +0000 2024-03-11T22:14:45.080 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-03-11 22:14:13.940790510 +0000 2024-03-11T22:14:45.081 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T22:14:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-03-11T22:14:45.152 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-03-11T22:14:45.152 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000255095 s, 2.0 MB/s 2024-03-11T22:14:45.153 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T22:14:45.210 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2024-03-11T22:14:45.270 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T22:14:45.270 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:14:45.271 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-11T22:14:45.271 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:14:45.271 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:14:45.271 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-03-11 22:14:35.518262165 +0000 2024-03-11T22:14:45.271 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-11 22:14:14.305781573 +0000 2024-03-11T22:14:45.271 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-11 22:14:14.305781573 +0000 2024-03-11T22:14:45.271 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-03-11 22:14:14.305781573 +0000 2024-03-11T22:14:45.271 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T22:14:45.341 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-03-11T22:14:45.342 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-03-11T22:14:45.342 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000207594 s, 2.5 MB/s 2024-03-11T22:14:45.343 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T22:14:45.399 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2024-03-11T22:14:45.454 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T22:14:45.454 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:14:45.455 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-11T22:14:45.455 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:14:45.455 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:14:45.455 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-03-11 22:14:35.518262165 +0000 2024-03-11T22:14:45.455 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-11 22:14:14.672772587 +0000 2024-03-11T22:14:45.455 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-11 22:14:14.672772587 +0000 2024-03-11T22:14:45.455 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-03-11 22:14:14.672772587 +0000 2024-03-11T22:14:45.455 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T22:14:45.525 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-03-11T22:14:45.525 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-03-11T22:14:45.525 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000208759 s, 2.5 MB/s 2024-03-11T22:14:45.526 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T22:14:45.582 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2024-03-11T22:14:45.637 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T22:14:45.637 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:14:45.637 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-11T22:14:45.637 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:14:45.638 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:14:45.638 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-03-11 22:14:35.518262165 +0000 2024-03-11T22:14:45.638 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-11 22:14:15.062763038 +0000 2024-03-11T22:14:45.638 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-11 22:14:15.062763038 +0000 2024-03-11T22:14:45.638 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-03-11 22:14:15.062763038 +0000 2024-03-11T22:14:45.638 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T22:14:45.707 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-03-11T22:14:45.707 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-03-11T22:14:45.707 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000183708 s, 2.8 MB/s 2024-03-11T22:14:45.708 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T22:14:45.765 DEBUG:teuthology.orchestra.run.smithi027:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-11T22:14:45.954 INFO:teuthology.orchestra.run.smithi027.stdout:loop 2024-03-11T22:14:45.955 INFO:tasks.nvme_loop:Connecting nvme_loop smithi027:/dev/vg_nvme/lv_1... 2024-03-11T22:14:45.955 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-11T22:14:46.002 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-03-11T22:14:46.049 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/vg_nvme/lv_11 2024-03-11T22:14:46.093 INFO:tasks.nvme_loop:Connecting nvme_loop smithi027:/dev/vg_nvme/lv_2... 2024-03-11T22:14:46.094 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-11T22:14:46.139 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-03-11T22:14:46.187 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/vg_nvme/lv_21 2024-03-11T22:14:46.229 INFO:tasks.nvme_loop:Connecting nvme_loop smithi027:/dev/vg_nvme/lv_3... 2024-03-11T22:14:46.230 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-11T22:14:46.282 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-03-11T22:14:46.331 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/vg_nvme/lv_31 2024-03-11T22:14:46.367 INFO:tasks.nvme_loop:Connecting nvme_loop smithi027:/dev/vg_nvme/lv_4... 2024-03-11T22:14:46.367 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-11T22:14:46.413 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-03-11T22:14:46.462 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/vg_nvme/lv_41 2024-03-11T22:14:46.498 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-11T22:14:46.498 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:14:46.516 DEBUG:teuthology.orchestra.run.smithi027:> sudo nvme list 2024-03-11T22:14:46.587 INFO:teuthology.orchestra.run.smithi027.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-11T22:14:46.587 INFO:teuthology.orchestra.run.smithi027.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-11T22:14:46.587 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/nvme0n1 CVFT5331001C400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-11T22:14:46.587 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/nvme1n1 98e758ac8b77779387f6 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:14:46.587 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/nvme2n1 a94b0ca315d15e9d7182 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:14:46.587 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/nvme3n1 5971608164b7de7dbd15 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:14:46.587 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/nvme4n1 d036649a413c9d693ff0 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:14:46.588 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T22:14:46.588 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-11T22:14:46.588 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/scratch_devs 2024-03-11T22:14:46.660 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-11T22:14:46.661 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:14:46.680 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T22:14:46.680 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_1 2024-03-11T22:14:46.737 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T22:14:46.738 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:14:46.738 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-11T22:14:46.738 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:14:46.738 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:14:46.738 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-11 22:14:35.516995062 +0000 2024-03-11T22:14:46.738 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-11 22:14:07.562370364 +0000 2024-03-11T22:14:46.738 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-11 22:14:07.562370364 +0000 2024-03-11T22:14:46.738 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-11 22:14:07.562370364 +0000 2024-03-11T22:14:46.738 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T22:14:46.809 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-11T22:14:46.809 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-11T22:14:46.809 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.00021603 s, 2.4 MB/s 2024-03-11T22:14:46.810 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T22:14:46.867 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_2 2024-03-11T22:14:46.923 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T22:14:46.924 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:14:46.924 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-11T22:14:46.924 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:14:46.924 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:14:46.924 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-11 22:14:35.516995062 +0000 2024-03-11T22:14:46.924 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-11 22:14:07.961365007 +0000 2024-03-11T22:14:46.924 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-11 22:14:07.961365007 +0000 2024-03-11T22:14:46.924 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-11 22:14:07.961365007 +0000 2024-03-11T22:14:46.924 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T22:14:46.992 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-11T22:14:46.992 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-11T22:14:46.993 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000239328 s, 2.1 MB/s 2024-03-11T22:14:46.994 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T22:14:47.052 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_3 2024-03-11T22:14:47.108 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T22:14:47.109 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:14:47.109 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-11T22:14:47.109 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:14:47.109 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:14:47.109 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-11 22:14:35.516995062 +0000 2024-03-11T22:14:47.109 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-11 22:14:08.312360295 +0000 2024-03-11T22:14:47.109 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-11 22:14:08.312360295 +0000 2024-03-11T22:14:47.109 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-11 22:14:08.312360295 +0000 2024-03-11T22:14:47.109 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T22:14:47.177 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-11T22:14:47.178 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-11T22:14:47.178 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000196201 s, 2.6 MB/s 2024-03-11T22:14:47.179 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T22:14:47.237 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_4 2024-03-11T22:14:47.293 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T22:14:47.293 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:14:47.293 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-11T22:14:47.294 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:14:47.294 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:14:47.294 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-11 22:14:35.516995062 +0000 2024-03-11T22:14:47.294 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-11 22:14:08.675355422 +0000 2024-03-11T22:14:47.294 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-11 22:14:08.675355422 +0000 2024-03-11T22:14:47.294 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-11 22:14:08.675355422 +0000 2024-03-11T22:14:47.294 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T22:14:47.360 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-11T22:14:47.360 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-11T22:14:47.360 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000218469 s, 2.3 MB/s 2024-03-11T22:14:47.361 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T22:14:47.419 DEBUG:teuthology.orchestra.run.smithi174:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-11T22:14:47.576 INFO:teuthology.orchestra.run.smithi174.stdout:loop 2024-03-11T22:14:47.577 INFO:tasks.nvme_loop:Connecting nvme_loop smithi174:/dev/vg_nvme/lv_1... 2024-03-11T22:14:47.577 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-11T22:14:47.615 INFO:teuthology.orchestra.run.smithi174.stdout:1 2024-03-11T22:14:47.653 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/vg_nvme/lv_11 2024-03-11T22:14:47.694 INFO:tasks.nvme_loop:Connecting nvme_loop smithi174:/dev/vg_nvme/lv_2... 2024-03-11T22:14:47.694 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-11T22:14:47.787 INFO:teuthology.orchestra.run.smithi174.stdout:1 2024-03-11T22:14:47.827 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/vg_nvme/lv_21 2024-03-11T22:14:47.856 INFO:tasks.nvme_loop:Connecting nvme_loop smithi174:/dev/vg_nvme/lv_3... 2024-03-11T22:14:47.856 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-11T22:14:47.898 INFO:teuthology.orchestra.run.smithi174.stdout:1 2024-03-11T22:14:47.937 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/vg_nvme/lv_31 2024-03-11T22:14:47.965 INFO:tasks.nvme_loop:Connecting nvme_loop smithi174:/dev/vg_nvme/lv_4... 2024-03-11T22:14:47.966 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-11T22:14:48.045 INFO:teuthology.orchestra.run.smithi174.stdout:1 2024-03-11T22:14:48.084 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/vg_nvme/lv_41 2024-03-11T22:14:48.114 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-11T22:14:48.115 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:14:48.171 DEBUG:teuthology.orchestra.run.smithi174:> sudo nvme list 2024-03-11T22:14:48.237 INFO:teuthology.orchestra.run.smithi174.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-11T22:14:48.237 INFO:teuthology.orchestra.run.smithi174.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-11T22:14:48.237 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/nvme0n1 PHFT62050074400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-11T22:14:48.237 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/nvme1n1 55849c08adb796cb573c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:14:48.237 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/nvme2n1 055b9a0c6f0dfd1cb5fe Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:14:48.238 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/nvme3n1 4bbd8d8a96512acc5609 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:14:48.238 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/nvme4n1 f941cc85995f17bbd413 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:14:48.238 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T22:14:48.239 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-11T22:14:48.239 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/scratch_devs 2024-03-11T22:14:48.306 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-11T22:14:48.306 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:14:48.325 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T22:14:48.325 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_1 2024-03-11T22:14:48.383 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T22:14:48.383 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:14:48.383 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-11T22:14:48.383 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:14:48.383 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:14:48.384 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-11 22:14:35.529295056 +0000 2024-03-11T22:14:48.384 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-11 22:14:07.526044137 +0000 2024-03-11T22:14:48.384 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-11 22:14:07.526044137 +0000 2024-03-11T22:14:48.384 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-11 22:14:07.526044137 +0000 2024-03-11T22:14:48.384 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T22:14:48.453 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-11T22:14:48.453 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-11T22:14:48.453 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000221679 s, 2.3 MB/s 2024-03-11T22:14:48.455 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T22:14:48.513 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_2 2024-03-11T22:14:48.570 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T22:14:48.570 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:14:48.570 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-11T22:14:48.570 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:14:48.570 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:14:48.570 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-11 22:14:35.529295056 +0000 2024-03-11T22:14:48.570 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-11 22:14:07.872034881 +0000 2024-03-11T22:14:48.570 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-11 22:14:07.872034881 +0000 2024-03-11T22:14:48.570 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-11 22:14:07.872034881 +0000 2024-03-11T22:14:48.571 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T22:14:48.638 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-11T22:14:48.638 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-11T22:14:48.638 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000226379 s, 2.3 MB/s 2024-03-11T22:14:48.639 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T22:14:48.698 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_3 2024-03-11T22:14:48.753 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T22:14:48.753 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:14:48.753 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-11T22:14:48.754 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:14:48.754 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:14:48.754 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-11 22:14:35.529295056 +0000 2024-03-11T22:14:48.754 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-11 22:14:08.232025250 +0000 2024-03-11T22:14:48.754 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-11 22:14:08.232025250 +0000 2024-03-11T22:14:48.754 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-11 22:14:08.232025250 +0000 2024-03-11T22:14:48.754 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T22:14:48.820 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-11T22:14:48.820 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-11T22:14:48.820 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000191475 s, 2.7 MB/s 2024-03-11T22:14:48.821 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T22:14:48.878 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_4 2024-03-11T22:14:48.933 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T22:14:48.933 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:14:48.934 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-11T22:14:48.934 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:14:48.934 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:14:48.934 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-11 22:14:35.529295056 +0000 2024-03-11T22:14:48.934 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-11 22:14:08.594015566 +0000 2024-03-11T22:14:48.934 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-11 22:14:08.594015566 +0000 2024-03-11T22:14:48.934 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-11 22:14:08.594015566 +0000 2024-03-11T22:14:48.935 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T22:14:49.000 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-11T22:14:49.000 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-11T22:14:49.000 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000189189 s, 2.7 MB/s 2024-03-11T22:14:49.001 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T22:14:49.056 DEBUG:teuthology.orchestra.run.smithi183:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-11T22:14:49.212 INFO:teuthology.orchestra.run.smithi183.stdout:loop 2024-03-11T22:14:49.214 INFO:tasks.nvme_loop:Connecting nvme_loop smithi183:/dev/vg_nvme/lv_1... 2024-03-11T22:14:49.214 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-11T22:14:49.252 INFO:teuthology.orchestra.run.smithi183.stdout:1 2024-03-11T22:14:49.290 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/vg_nvme/lv_11 2024-03-11T22:14:49.321 INFO:tasks.nvme_loop:Connecting nvme_loop smithi183:/dev/vg_nvme/lv_2... 2024-03-11T22:14:49.321 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-11T22:14:49.402 INFO:teuthology.orchestra.run.smithi183.stdout:1 2024-03-11T22:14:49.441 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/vg_nvme/lv_21 2024-03-11T22:14:49.479 INFO:tasks.nvme_loop:Connecting nvme_loop smithi183:/dev/vg_nvme/lv_3... 2024-03-11T22:14:49.479 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-11T22:14:49.559 INFO:teuthology.orchestra.run.smithi183.stdout:1 2024-03-11T22:14:49.598 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/vg_nvme/lv_31 2024-03-11T22:14:49.628 INFO:tasks.nvme_loop:Connecting nvme_loop smithi183:/dev/vg_nvme/lv_4... 2024-03-11T22:14:49.628 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-11T22:14:49.706 INFO:teuthology.orchestra.run.smithi183.stdout:1 2024-03-11T22:14:49.745 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/vg_nvme/lv_41 2024-03-11T22:14:49.786 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-11T22:14:49.786 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:14:49.841 DEBUG:teuthology.orchestra.run.smithi183:> sudo nvme list 2024-03-11T22:14:49.906 INFO:teuthology.orchestra.run.smithi183.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-11T22:14:49.907 INFO:teuthology.orchestra.run.smithi183.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-11T22:14:49.907 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/nvme0n1 PHFT620500QJ400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-11T22:14:49.907 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/nvme1n1 55f82714da76bf163c37 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:14:49.907 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/nvme2n1 1c4319991c9dd16ce489 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:14:49.907 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/nvme3n1 048f6294288f52b78793 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:14:49.907 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/nvme4n1 96ee08236bd417d0c5f4 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:14:49.908 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T22:14:49.908 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-11T22:14:49.908 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/scratch_devs 2024-03-11T22:14:49.975 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-11T22:14:50.081 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'global': {'mon election default strategy': 3}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'} 2024-03-11T22:14:50.081 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:14:50.081 INFO:tasks.cephadm:Cluster fsid is c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:14:50.081 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-11T22:14:50.082 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.27', 'mon.b': '172.21.15.174', 'mon.c': '172.21.15.183'} 2024-03-11T22:14:50.082 INFO:tasks.cephadm:First mon is mon.a on smithi027 2024-03-11T22:14:50.082 INFO:tasks.cephadm:First mgr is y 2024-03-11T22:14:50.082 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-11T22:14:50.082 DEBUG:teuthology.orchestra.run.smithi027:> sudo hostname $(hostname -s) 2024-03-11T22:14:50.120 DEBUG:teuthology.orchestra.run.smithi174:> sudo hostname $(hostname -s) 2024-03-11T22:14:50.153 DEBUG:teuthology.orchestra.run.smithi183:> sudo hostname $(hostname -s) 2024-03-11T22:14:50.180 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-11T22:14:50.181 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:14:50.341 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77431/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:08:19.449750', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77429/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:06:59.696700', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-11T22:14:50.488 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 94a539707733325322964c6a187c6a1b41be1daf from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:14:50.489 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm 2024-03-11T22:14:50.490 DEBUG:teuthology.orchestra.run.smithi027:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T22:14:51.171 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 22:14 /home/ubuntu/cephtest/cephadm 2024-03-11T22:14:51.172 DEBUG:teuthology.orchestra.run.smithi174:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T22:14:51.649 INFO:teuthology.orchestra.run.smithi174.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 22:14 /home/ubuntu/cephtest/cephadm 2024-03-11T22:14:51.649 DEBUG:teuthology.orchestra.run.smithi183:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T22:14:52.119 INFO:teuthology.orchestra.run.smithi183.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 22:14 /home/ubuntu/cephtest/cephadm 2024-03-11T22:14:52.120 DEBUG:teuthology.orchestra.run.smithi027:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T22:14:52.140 DEBUG:teuthology.orchestra.run.smithi174:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T22:14:52.159 DEBUG:teuthology.orchestra.run.smithi183:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T22:14:52.191 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf on all hosts... 2024-03-11T22:14:52.191 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T22:14:52.194 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T22:14:52.203 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T22:14:52.992 INFO:teuthology.orchestra.run.smithi027.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T22:14:53.008 INFO:teuthology.orchestra.run.smithi183.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T22:14:53.088 INFO:teuthology.orchestra.run.smithi174.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T22:15:31.876 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-11T22:15:31.876 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T22:15:31.876 INFO:teuthology.orchestra.run.smithi027.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T22:15:31.877 INFO:teuthology.orchestra.run.smithi027.stdout: "repo_digests": [ 2024-03-11T22:15:31.877 INFO:teuthology.orchestra.run.smithi027.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T22:15:31.877 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-03-11T22:15:31.877 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-11T22:15:38.827 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-11T22:15:38.827 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T22:15:38.827 INFO:teuthology.orchestra.run.smithi174.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T22:15:38.827 INFO:teuthology.orchestra.run.smithi174.stdout: "repo_digests": [ 2024-03-11T22:15:38.827 INFO:teuthology.orchestra.run.smithi174.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T22:15:38.827 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-11T22:15:38.827 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-11T22:15:41.769 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-03-11T22:15:41.769 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T22:15:41.769 INFO:teuthology.orchestra.run.smithi183.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T22:15:41.769 INFO:teuthology.orchestra.run.smithi183.stdout: "repo_digests": [ 2024-03-11T22:15:41.770 INFO:teuthology.orchestra.run.smithi183.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T22:15:41.770 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-03-11T22:15:41.770 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-03-11T22:15:41.796 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph 2024-03-11T22:15:41.838 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /etc/ceph 2024-03-11T22:15:41.873 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph 2024-03-11T22:15:41.905 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /etc/ceph 2024-03-11T22:15:41.938 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 777 /etc/ceph 2024-03-11T22:15:41.969 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 777 /etc/ceph 2024-03-11T22:15:41.997 INFO:tasks.cephadm:Writing seed config... 2024-03-11T22:15:41.999 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-11T22:15:41.999 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-11T22:15:41.999 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-03-11T22:15:41.999 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-03-11T22:15:41.999 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-11T22:15:42.000 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-11T22:15:42.000 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-11T22:15:42.000 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-11T22:15:42.000 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-11T22:15:42.000 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-11T22:15:42.001 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-11T22:15:42.001 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-11T22:15:42.019 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c71f9f14-dff4-11ee-95c8-87774f69a715 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-11T22:15:42.020 DEBUG:teuthology.orchestra.run.smithi027:mon.a> sudo journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mon.a.service 2024-03-11T22:15:42.063 DEBUG:teuthology.orchestra.run.smithi027:mgr.y> sudo journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mgr.y.service 2024-03-11T22:15:42.106 INFO:tasks.cephadm:Bootstrapping... 2024-03-11T22:15:42.107 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf -v bootstrap --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.27 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:15:42.308 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:15:42.308 INFO:teuthology.orchestra.run.smithi027.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf', '-v', 'bootstrap', '--fsid', 'c71f9f14-dff4-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.27', '--skip-admin-label'] 2024-03-11T22:15:42.334 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T22:15:42.335 INFO:teuthology.orchestra.run.smithi027.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-11T22:15:42.335 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying podman|docker is present... 2024-03-11T22:15:42.358 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T22:15:42.359 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying lvm2 is present... 2024-03-11T22:15:42.359 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying time synchronization is in place... 2024-03-11T22:15:42.366 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T22:15:42.366 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T22:15:42.373 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T22:15:42.373 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-03-11T22:15:42.380 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout enabled 2024-03-11T22:15:42.387 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout active 2024-03-11T22:15:42.387 INFO:teuthology.orchestra.run.smithi027.stdout:Unit chronyd.service is enabled and running 2024-03-11T22:15:42.387 INFO:teuthology.orchestra.run.smithi027.stdout:Repeating the final host check... 2024-03-11T22:15:42.412 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T22:15:42.413 INFO:teuthology.orchestra.run.smithi027.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-11T22:15:42.413 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl is present 2024-03-11T22:15:42.413 INFO:teuthology.orchestra.run.smithi027.stdout:lvcreate is present 2024-03-11T22:15:42.419 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T22:15:42.419 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T22:15:42.427 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T22:15:42.427 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-03-11T22:15:42.434 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout enabled 2024-03-11T22:15:42.440 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout active 2024-03-11T22:15:42.440 INFO:teuthology.orchestra.run.smithi027.stdout:Unit chronyd.service is enabled and running 2024-03-11T22:15:42.440 INFO:teuthology.orchestra.run.smithi027.stdout:Host looks OK 2024-03-11T22:15:42.441 INFO:teuthology.orchestra.run.smithi027.stdout:Cluster fsid: c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:15:42.441 INFO:teuthology.orchestra.run.smithi027.stdout:Acquiring lock 139998456276016 on /run/cephadm/c71f9f14-dff4-11ee-95c8-87774f69a715.lock 2024-03-11T22:15:42.441 INFO:teuthology.orchestra.run.smithi027.stdout:Lock 139998456276016 acquired on /run/cephadm/c71f9f14-dff4-11ee-95c8-87774f69a715.lock 2024-03-11T22:15:42.441 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying IP 172.21.15.27 port 3300 ... 2024-03-11T22:15:42.441 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying IP 172.21.15.27 port 6789 ... 2024-03-11T22:15:42.441 INFO:teuthology.orchestra.run.smithi027.stdout:Base mon IP(s) is [172.21.15.27:3300, 172.21.15.27:6789], mon addrv is [v2:172.21.15.27:3300,v1:172.21.15.27:6789] 2024-03-11T22:15:42.444 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.27 metric 100 2024-03-11T22:15:42.444 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.27 metric 100 2024-03-11T22:15:42.447 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-11T22:15:42.447 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-11T22:15:42.447 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-11T22:15:42.449 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-11T22:15:42.449 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-11T22:15:42.449 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T22:15:42.449 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-11T22:15:42.450 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-03-11T22:15:42.450 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T22:15:42.450 INFO:teuthology.orchestra.run.smithi027.stdout:Mon IP `172.21.15.27` is in CIDR network `172.21.0.0/20` 2024-03-11T22:15:42.451 INFO:teuthology.orchestra.run.smithi027.stdout:Mon IP `172.21.15.27` is in CIDR network `172.21.0.0/20` 2024-03-11T22:15:42.451 INFO:teuthology.orchestra.run.smithi027.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-11T22:15:42.451 INFO:teuthology.orchestra.run.smithi027.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-11T22:15:42.451 INFO:teuthology.orchestra.run.smithi027.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T22:15:42.856 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T22:15:42.857 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T22:15:42.857 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Getting image source signatures 2024-03-11T22:15:42.857 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying blob sha256:0b3c7bee4e66dab4b5376aaf0baf8ba5fc1c3e66c27dc601157782d617bd9dc6 2024-03-11T22:15:42.857 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-11T22:15:42.857 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying config sha256:a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T22:15:42.857 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-11T22:15:43.994 INFO:teuthology.orchestra.run.smithi027.stdout:ceph: stdout ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-11T22:15:43.994 INFO:teuthology.orchestra.run.smithi027.stdout:Ceph version: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-11T22:15:43.994 INFO:teuthology.orchestra.run.smithi027.stdout:Extracting ceph user uid/gid from container image... 2024-03-11T22:15:44.858 INFO:teuthology.orchestra.run.smithi027.stdout:stat: stdout 167 167 2024-03-11T22:15:44.858 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial keys... 2024-03-11T22:15:45.746 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQCRgu9lKIYrDRAAbR5q/C0kMZAUJt49V3uQrQ== 2024-03-11T22:15:46.602 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQCSgu9lla2NBBAAzfT4biYcVkbufbzCq71Yuw== 2024-03-11T22:15:51.054 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQCSgu9lTSgIOBAA1TtrAxU73Up9vCcdszHX8g== 2024-03-11T22:15:51.055 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial monmap... 2024-03-11T22:15:55.384 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T22:15:55.384 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-11T22:15:55.384 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:15:55.384 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T22:15:55.384 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool for a [v2:172.21.15.27:3300,v1:172.21.15.27:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T22:15:55.384 INFO:teuthology.orchestra.run.smithi027.stdout:setting min_mon_release = quincy 2024-03-11T22:15:55.384 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: set fsid to c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:15:55.384 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T22:15:55.384 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:15:55.384 INFO:teuthology.orchestra.run.smithi027.stdout:Creating mon... 2024-03-11T22:15:56.352 INFO:teuthology.orchestra.run.smithi027.stdout:create mon.a on 2024-03-11T22:15:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-11T22:15:56.964 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c71f9f14-dff4-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-c71f9f14-dff4-11ee-95c8-87774f69a715.target. 2024-03-11T22:15:56.964 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-c71f9f14-dff4-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-c71f9f14-dff4-11ee-95c8-87774f69a715.target. 2024-03-11T22:15:57.203 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mon.a 2024-03-11T22:15:57.204 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-11T22:15:57.378 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c71f9f14-dff4-11ee-95c8-87774f69a715.target.wants/ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@.service. 2024-03-11T22:15:57.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:15:57 smithi027 podman[25272]: 2024-03-11 22:15:57.522900664 +0000 UTC m=+0.018124801 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:15:57.876 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T22:15:57.876 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-03-11T22:15:57.883 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T22:15:57.883 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-03-11T22:15:57.883 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-03-11T22:15:57.883 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T22:15:57.884 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mon to start... 2024-03-11T22:15:57.884 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mon... 2024-03-11T22:15:58.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:15:57 smithi027 podman[25272]: 2024-03-11 22:15:57.764933421 +0000 UTC m=+0.260157565 container init 34657059aff25b6d9775aa6cd41f349a5c0ed828835a5ebcaaf7a38c7394c0e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:15:58.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:15:57 smithi027 podman[25272]: 2024-03-11 22:15:57.770663747 +0000 UTC m=+0.265887882 container start 34657059aff25b6d9775aa6cd41f349a5c0ed828835a5ebcaaf7a38c7394c0e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397) 2024-03-11T22:15:58.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:15:57 smithi027 ceph-mon[25310]: mkfs c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:15:58.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:15:57 smithi027 ceph-mon[25310]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T22:15:58.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:15:57 smithi027 bash[25272]: 34657059aff25b6d9775aa6cd41f349a5c0ed828835a5ebcaaf7a38c7394c0e2 2024-03-11T22:15:58.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:15:57 smithi027 systemd[1]: Started Ceph mon.a for c71f9f14-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:15:59.261 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:15:58 smithi027 ceph-mon[25310]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T22:15:59.261 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:15:58 smithi027 ceph-mon[25310]: monmap e1: 1 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:15:59.261 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:15:58 smithi027 ceph-mon[25310]: fsmap 2024-03-11T22:15:59.261 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:15:58 smithi027 ceph-mon[25310]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T22:15:59.261 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:15:58 smithi027 ceph-mon[25310]: mgrmap e1: no daemons active 2024-03-11T22:15:59.262 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:15:58 smithi027 ceph-mon[25310]: from='client.? 172.21.15.27:0/2429950269' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-11T22:15:59.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout cluster: 2024-03-11T22:15:59.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout id: c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:15:59.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-11T22:15:59.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-03-11T22:15:59.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout services: 2024-03-11T22:15:59.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.607073s) 2024-03-11T22:15:59.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-11T22:15:59.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-11T22:15:59.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-03-11T22:15:59.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout data: 2024-03-11T22:15:59.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-11T22:15:59.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-11T22:15:59.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-11T22:15:59.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout pgs: 2024-03-11T22:15:59.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-03-11T22:15:59.267 INFO:teuthology.orchestra.run.smithi027.stdout:mon is available 2024-03-11T22:15:59.267 INFO:teuthology.orchestra.run.smithi027.stdout:Assimilating anything we can from ceph.conf... 2024-03-11T22:15:59.895 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:15:59 smithi027 ceph-mon[25310]: from='client.? 172.21.15.27:0/3083093412' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T22:15:59.895 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:15:59 smithi027 ceph-mon[25310]: from='client.? 172.21.15.27:0/3083093412' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T22:16:01.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-03-11T22:16:01.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T22:16:01.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout fsid = c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:16:01.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.27:3300,v1:172.21.15.27:6789] 2024-03-11T22:16:01.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T22:16:01.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T22:16:01.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T22:16:01.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T22:16:01.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-03-11T22:16:01.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T22:16:01.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = False 2024-03-11T22:16:01.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T22:16:01.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-03-11T22:16:01.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T22:16:01.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T22:16:01.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T22:16:01.254 INFO:teuthology.orchestra.run.smithi027.stdout:Generating new minimal ceph.conf... 2024-03-11T22:16:02.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:01 smithi027 ceph-mon[25310]: from='client.? 172.21.15.27:0/4029382780' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:16:02.357 INFO:teuthology.orchestra.run.smithi027.stdout:Restarting the monitor... 2024-03-11T22:16:02.636 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:02 smithi027 systemd[1]: Stopping Ceph mon.a for c71f9f14-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:16:03.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:02 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-a[25286]: 2024-03-11T22:16:02.635+0000 7f4534eba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:16:03.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:02 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-a[25286]: 2024-03-11T22:16:02.635+0000 7f4534eba700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-11T22:16:03.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:02 smithi027 podman[25646]: 2024-03-11 22:16:02.759841157 +0000 UTC m=+0.232607234 container died 34657059aff25b6d9775aa6cd41f349a5c0ed828835a5ebcaaf7a38c7394c0e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-11T22:16:03.258 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 podman[25646]: 2024-03-11 22:16:03.124693399 +0000 UTC m=+0.597459484 container cleanup 34657059aff25b6d9775aa6cd41f349a5c0ed828835a5ebcaaf7a38c7394c0e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-a, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-11T22:16:03.259 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 bash[25646]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-a 2024-03-11T22:16:03.259 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 podman[25665]: 2024-03-11 22:16:03.258979955 +0000 UTC m=+0.493845042 container remove 34657059aff25b6d9775aa6cd41f349a5c0ed828835a5ebcaaf7a38c7394c0e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-a, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:16:03.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 systemd[1]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-11T22:16:03.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 systemd[1]: Stopped Ceph mon.a for c71f9f14-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:16:03.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 systemd[1]: Starting Ceph mon.a for c71f9f14-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:16:03.961 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 podman[25759]: 2024-03-11 22:16:03.583241907 +0000 UTC m=+0.018623511 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:16:03.961 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 podman[25759]: 2024-03-11 22:16:03.701557044 +0000 UTC m=+0.136938671 container create 77695fc9e008d15a63bfed6f337637aaeabadde59a3dd7e6ff22bcd268ed65ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-a, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=) 2024-03-11T22:16:03.961 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 podman[25759]: 2024-03-11 22:16:03.878766488 +0000 UTC m=+0.314148089 container init 77695fc9e008d15a63bfed6f337637aaeabadde59a3dd7e6ff22bcd268ed65ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-a, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0) 2024-03-11T22:16:03.961 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 podman[25759]: 2024-03-11 22:16:03.884200581 +0000 UTC m=+0.319582166 container start 77695fc9e008d15a63bfed6f337637aaeabadde59a3dd7e6ff22bcd268ed65ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-a, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-11T22:16:03.961 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T22:16:03.961 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 7 2024-03-11T22:16:03.961 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: pidfile_write: ignore empty --pid-file 2024-03-11T22:16:03.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: load: jerasure load: lrc 2024-03-11T22:16:03.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: RocksDB version: 7.9.2 2024-03-11T22:16:03.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Git sha 0 2024-03-11T22:16:03.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T22:16:03.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: DB SUMMARY 2024-03-11T22:16:03.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: DB Session ID: Z6GT8982SKZ5LHFZ2M16 2024-03-11T22:16:03.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: CURRENT file: CURRENT 2024-03-11T22:16:03.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T22:16:03.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-11T22:16:03.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-11T22:16:03.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84629 ; 2024-03-11T22:16:03.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.error_if_exists: 0 2024-03-11T22:16:03.963 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.create_if_missing: 0 2024-03-11T22:16:03.963 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.paranoid_checks: 1 2024-03-11T22:16:03.963 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T22:16:03.963 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T22:16:03.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T22:16:03.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.env: 0x558bf5b56ae0 2024-03-11T22:16:03.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T22:16:03.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.info_log: 0x558bf64418c0 2024-03-11T22:16:03.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T22:16:03.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.statistics: (nil) 2024-03-11T22:16:03.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.use_fsync: 0 2024-03-11T22:16:03.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_log_file_size: 0 2024-03-11T22:16:03.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T22:16:03.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T22:16:03.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T22:16:03.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T22:16:03.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.allow_fallocate: 1 2024-03-11T22:16:03.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T22:16:03.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T22:16:03.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.use_direct_reads: 0 2024-03-11T22:16:03.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T22:16:03.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T22:16:03.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.db_log_dir: 2024-03-11T22:16:03.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.wal_dir: 2024-03-11T22:16:03.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T22:16:03.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T22:16:03.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T22:16:03.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T22:16:03.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T22:16:03.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T22:16:03.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T22:16:03.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T22:16:03.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.write_buffer_manager: 0x558bf6415400 2024-03-11T22:16:03.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T22:16:03.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T22:16:03.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T22:16:03.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T22:16:03.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T22:16:03.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T22:16:03.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T22:16:03.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T22:16:03.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.unordered_write: 0 2024-03-11T22:16:03.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T22:16:03.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T22:16:03.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T22:16:03.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T22:16:03.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.row_cache: None 2024-03-11T22:16:03.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.wal_filter: None 2024-03-11T22:16:03.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T22:16:03.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T22:16:03.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.two_write_queues: 0 2024-03-11T22:16:03.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T22:16:03.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.wal_compression: 0 2024-03-11T22:16:03.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.atomic_flush: 0 2024-03-11T22:16:03.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T22:16:03.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T22:16:03.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T22:16:03.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.log_readahead_size: 0 2024-03-11T22:16:03.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T22:16:03.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T22:16:03.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T22:16:03.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T22:16:03.968 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T22:16:03.968 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T22:16:03.968 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T22:16:03.968 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_background_jobs: 2 2024-03-11T22:16:03.968 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_background_compactions: -1 2024-03-11T22:16:03.968 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_subcompactions: 1 2024-03-11T22:16:03.968 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T22:16:03.968 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T22:16:03.968 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T22:16:03.968 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T22:16:03.968 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T22:16:03.968 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T22:16:03.968 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T22:16:03.968 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T22:16:03.968 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_open_files: -1 2024-03-11T22:16:03.968 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T22:16:03.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T22:16:03.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T22:16:03.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T22:16:03.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_background_flushes: -1 2024-03-11T22:16:03.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Compression algorithms supported: 2024-03-11T22:16:03.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T22:16:03.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: kZSTD supported: 0 2024-03-11T22:16:03.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: kXpressCompression supported: 0 2024-03-11T22:16:03.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T22:16:03.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: kZlibCompression supported: 1 2024-03-11T22:16:03.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: kSnappyCompression supported: 1 2024-03-11T22:16:03.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: kLZ4Compression supported: 1 2024-03-11T22:16:03.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: kBZip2Compression supported: 0 2024-03-11T22:16:03.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T22:16:03.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T22:16:03.970 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-11T22:16:03.970 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T22:16:03.970 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T22:16:03.970 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.merge_operator: 2024-03-11T22:16:03.970 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compaction_filter: None 2024-03-11T22:16:03.970 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T22:16:03.970 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T22:16:03.970 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T22:16:03.970 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T22:16:03.970 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558bf64419c0) 2024-03-11T22:16:03.970 INFO:journalctl@ceph.mon.a.smithi027.stdout: cache_index_and_filter_blocks: 1 2024-03-11T22:16:03.970 INFO:journalctl@ceph.mon.a.smithi027.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T22:16:03.970 INFO:journalctl@ceph.mon.a.smithi027.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T22:16:03.970 INFO:journalctl@ceph.mon.a.smithi027.stdout: pin_top_level_index_and_filter: 1 2024-03-11T22:16:03.971 INFO:journalctl@ceph.mon.a.smithi027.stdout: index_type: 0 2024-03-11T22:16:03.971 INFO:journalctl@ceph.mon.a.smithi027.stdout: data_block_index_type: 0 2024-03-11T22:16:03.971 INFO:journalctl@ceph.mon.a.smithi027.stdout: index_shortening: 1 2024-03-11T22:16:03.971 INFO:journalctl@ceph.mon.a.smithi027.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T22:16:03.971 INFO:journalctl@ceph.mon.a.smithi027.stdout: checksum: 4 2024-03-11T22:16:03.971 INFO:journalctl@ceph.mon.a.smithi027.stdout: no_block_cache: 0 2024-03-11T22:16:03.971 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache: 0x558bf6445090 2024-03-11T22:16:03.971 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache_name: BinnedLRUCache 2024-03-11T22:16:03.971 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache_options: 2024-03-11T22:16:03.971 INFO:journalctl@ceph.mon.a.smithi027.stdout: capacity : 536870912 2024-03-11T22:16:03.971 INFO:journalctl@ceph.mon.a.smithi027.stdout: num_shard_bits : 4 2024-03-11T22:16:03.971 INFO:journalctl@ceph.mon.a.smithi027.stdout: strict_capacity_limit : 0 2024-03-11T22:16:03.971 INFO:journalctl@ceph.mon.a.smithi027.stdout: high_pri_pool_ratio: 0.000 2024-03-11T22:16:03.971 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache_compressed: (nil) 2024-03-11T22:16:03.971 INFO:journalctl@ceph.mon.a.smithi027.stdout: persistent_cache: (nil) 2024-03-11T22:16:03.971 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_size: 4096 2024-03-11T22:16:03.971 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_size_deviation: 10 2024-03-11T22:16:03.972 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_restart_interval: 16 2024-03-11T22:16:03.972 INFO:journalctl@ceph.mon.a.smithi027.stdout: index_block_restart_interval: 1 2024-03-11T22:16:03.972 INFO:journalctl@ceph.mon.a.smithi027.stdout: metadata_block_size: 4096 2024-03-11T22:16:03.972 INFO:journalctl@ceph.mon.a.smithi027.stdout: partition_filters: 0 2024-03-11T22:16:03.972 INFO:journalctl@ceph.mon.a.smithi027.stdout: use_delta_encoding: 1 2024-03-11T22:16:03.972 INFO:journalctl@ceph.mon.a.smithi027.stdout: filter_policy: bloomfilter 2024-03-11T22:16:03.972 INFO:journalctl@ceph.mon.a.smithi027.stdout: whole_key_filtering: 1 2024-03-11T22:16:03.972 INFO:journalctl@ceph.mon.a.smithi027.stdout: verify_compression: 0 2024-03-11T22:16:03.972 INFO:journalctl@ceph.mon.a.smithi027.stdout: read_amp_bytes_per_bit: 0 2024-03-11T22:16:03.972 INFO:journalctl@ceph.mon.a.smithi027.stdout: format_version: 5 2024-03-11T22:16:03.972 INFO:journalctl@ceph.mon.a.smithi027.stdout: enable_index_compression: 1 2024-03-11T22:16:03.972 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_align: 0 2024-03-11T22:16:03.972 INFO:journalctl@ceph.mon.a.smithi027.stdout: max_auto_readahead_size: 262144 2024-03-11T22:16:03.972 INFO:journalctl@ceph.mon.a.smithi027.stdout: prepopulate_block_cache: 0 2024-03-11T22:16:03.972 INFO:journalctl@ceph.mon.a.smithi027.stdout: initial_auto_readahead_size: 8192 2024-03-11T22:16:03.972 INFO:journalctl@ceph.mon.a.smithi027.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T22:16:03.972 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T22:16:03.973 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T22:16:03.973 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compression: NoCompression 2024-03-11T22:16:03.973 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T22:16:03.973 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T22:16:03.973 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T22:16:03.973 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.num_levels: 7 2024-03-11T22:16:03.973 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T22:16:03.973 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T22:16:03.973 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T22:16:03.973 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T22:16:03.973 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T22:16:03.973 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T22:16:03.973 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T22:16:03.973 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:16:03.973 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T22:16:03.973 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T22:16:03.974 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:16:03.974 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:16:03.974 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T22:16:03.974 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T22:16:03.974 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T22:16:03.974 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T22:16:03.974 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:16:03.974 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:16:03.974 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T22:16:03.974 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T22:16:03.974 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:16:03.974 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T22:16:03.974 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T22:16:03.974 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T22:16:03.974 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T22:16:03.975 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T22:16:03.975 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T22:16:03.975 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T22:16:03.975 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T22:16:03.975 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T22:16:03.975 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T22:16:03.975 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T22:16:03.975 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T22:16:03.975 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T22:16:03.975 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T22:16:03.975 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T22:16:03.975 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T22:16:03.975 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T22:16:03.975 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T22:16:03.975 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T22:16:03.975 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T22:16:03.976 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T22:16:03.976 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T22:16:03.976 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T22:16:03.976 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T22:16:03.976 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T22:16:03.976 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T22:16:03.976 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T22:16:03.976 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T22:16:03.976 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T22:16:03.976 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T22:16:03.976 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T22:16:03.976 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T22:16:03.977 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.table_properties_collectors: 2024-03-11T22:16:03.977 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.inplace_update_support: 0 2024-03-11T22:16:03.977 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T22:16:03.977 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T22:16:03.977 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T22:16:03.977 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T22:16:03.978 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.bloom_locality: 0 2024-03-11T22:16:03.978 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.max_successive_merges: 0 2024-03-11T22:16:03.978 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T22:16:03.978 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T22:16:03.978 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T22:16:03.978 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T22:16:03.978 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.ttl: 2592000 2024-03-11T22:16:03.978 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T22:16:03.978 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T22:16:03.978 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T22:16:03.978 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.enable_blob_files: false 2024-03-11T22:16:03.979 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.min_blob_size: 0 2024-03-11T22:16:03.979 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T22:16:03.979 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T22:16:03.979 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T22:16:03.979 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T22:16:03.979 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T22:16:03.979 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T22:16:03.979 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T22:16:03.979 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T22:16:03.979 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-11T22:16:03.979 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-11T22:16:03.979 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d5e06bca-53f2-4fc5-9f73-f2c7925c35df 2024-03-11T22:16:03.979 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710195363913728, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-11T22:16:03.979 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-11T22:16:03.979 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710195363914607, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81254, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79441, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73750, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710195363, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d5e06bca-53f2-4fc5-9f73-f2c7925c35df", "db_session_id": "Z6GT8982SKZ5LHFZ2M16", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-11T22:16:03.980 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710195363914667, "job": 1, "event": "recovery_finished"} 2024-03-11T22:16:03.980 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-11T22:16:03.980 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T22:16:03.980 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558bf6536000 2024-03-11T22:16:03.980 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: DB pointer 0x558bf6520000 2024-03-11T22:16:03.980 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T22:16:03.980 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T22:16:03.980 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** DB Stats ** 2024-03-11T22:16:03.980 INFO:journalctl@ceph.mon.a.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:16:03.980 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T22:16:03.980 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:16:03.980 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:16:03.980 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T22:16:03.980 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:16:03.980 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:16:03.980 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-03-11T22:16:03.981 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** Compaction Stats [default] ** 2024-03-11T22:16:03.981 INFO:journalctl@ceph.mon.a.smithi027.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:16:03.981 INFO:journalctl@ceph.mon.a.smithi027.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T22:16:03.981 INFO:journalctl@ceph.mon.a.smithi027.stdout: L0 2/0 81.18 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T22:16:03.981 INFO:journalctl@ceph.mon.a.smithi027.stdout: Sum 2/0 81.18 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T22:16:03.981 INFO:journalctl@ceph.mon.a.smithi027.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T22:16:03.981 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-03-11T22:16:03.981 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** Compaction Stats [default] ** 2024-03-11T22:16:03.981 INFO:journalctl@ceph.mon.a.smithi027.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:16:03.981 INFO:journalctl@ceph.mon.a.smithi027.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:16:03.981 INFO:journalctl@ceph.mon.a.smithi027.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 141.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T22:16:03.981 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-03-11T22:16:03.981 INFO:journalctl@ceph.mon.a.smithi027.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T22:16:03.981 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-03-11T22:16:03.981 INFO:journalctl@ceph.mon.a.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:16:03.982 INFO:journalctl@ceph.mon.a.smithi027.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T22:16:03.982 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T22:16:03.982 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T22:16:03.982 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T22:16:03.982 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T22:16:03.982 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative compaction: 0.00 GB write, 25.73 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:16:03.982 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval compaction: 0.00 GB write, 25.73 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:16:03.982 INFO:journalctl@ceph.mon.a.smithi027.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T22:16:03.982 INFO:journalctl@ceph.mon.a.smithi027.stdout: Block cache BinnedLRUCache@0x558bf6445090#7 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-11T22:16:03.982 INFO:journalctl@ceph.mon.a.smithi027.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-11T22:16:03.982 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-03-11T22:16:03.982 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T22:16:03.982 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: starting mon.a rank 0 at public addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] at bind addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:16:03.982 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: mon.a@-1(???) e1 preinit fsid c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:16:03.982 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: mon.a@-1(???).mds e1 new map 2024-03-11T22:16:03.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: mon.a@-1(???).mds e1 print_map 2024-03-11T22:16:03.983 INFO:journalctl@ceph.mon.a.smithi027.stdout: e1 2024-03-11T22:16:03.983 INFO:journalctl@ceph.mon.a.smithi027.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T22:16:03.983 INFO:journalctl@ceph.mon.a.smithi027.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T22:16:03.983 INFO:journalctl@ceph.mon.a.smithi027.stdout: legacy client fscid: -1 2024-03-11T22:16:03.983 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-03-11T22:16:03.983 INFO:journalctl@ceph.mon.a.smithi027.stdout: No filesystems configured 2024-03-11T22:16:03.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T22:16:03.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:16:03.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:16:03.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:16:03.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-11T22:16:03.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 bash[25759]: 77695fc9e008d15a63bfed6f337637aaeabadde59a3dd7e6ff22bcd268ed65ef 2024-03-11T22:16:03.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 systemd[1]: Started Ceph mon.a for c71f9f14-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:16:03.983 INFO:teuthology.orchestra.run.smithi027.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-11T22:16:04.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T22:16:04.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: monmap e1: 1 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:16:04.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: fsmap 2024-03-11T22:16:04.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T22:16:04.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:03 smithi027 ceph-mon[25797]: mgrmap e1: no daemons active 2024-03-11T22:16:04.995 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-11T22:16:05.016 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:16:05.016 INFO:teuthology.orchestra.run.smithi027.stdout:Creating mgr... 2024-03-11T22:16:05.017 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-11T22:16:05.017 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-11T22:16:05.300 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mgr.y 2024-03-11T22:16:05.300 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mgr.y.service: Unit ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mgr.y.service not loaded. 2024-03-11T22:16:05.466 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c71f9f14-dff4-11ee-95c8-87774f69a715.target.wants/ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@.service. 2024-03-11T22:16:05.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:05 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/1003899614' entity='client.admin' 2024-03-11T22:16:05.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:05 smithi027 podman[26039]: 2024-03-11 22:16:05.703448888 +0000 UTC m=+0.116206889 container create bd177f900fda5bd7c3d2e597984f2aa220de753a4880efa9d489747d986c15d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-11T22:16:06.018 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T22:16:06.018 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-03-11T22:16:06.025 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T22:16:06.026 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-03-11T22:16:06.026 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-03-11T22:16:06.026 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T22:16:06.034 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T22:16:06.035 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-03-11T22:16:06.042 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T22:16:06.042 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-03-11T22:16:06.042 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-03-11T22:16:06.042 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-11T22:16:06.042 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr to start... 2024-03-11T22:16:06.042 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr... 2024-03-11T22:16:06.167 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:05 smithi027 podman[26039]: 2024-03-11 22:16:05.905594247 +0000 UTC m=+0.318352265 container init bd177f900fda5bd7c3d2e597984f2aa220de753a4880efa9d489747d986c15d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y, GIT_CLEAN=True, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-11T22:16:06.167 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:05 smithi027 podman[26039]: 2024-03-11 22:16:05.911095932 +0000 UTC m=+0.323853934 container start bd177f900fda5bd7c3d2e597984f2aa220de753a4880efa9d489747d986c15d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-11T22:16:06.167 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:06 smithi027 bash[26039]: bd177f900fda5bd7c3d2e597984f2aa220de753a4880efa9d489747d986c15d6 2024-03-11T22:16:06.168 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:06 smithi027 systemd[1]: Started Ceph mgr.y for c71f9f14-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:16:06.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:06 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:06.476+0000 7f08f3818200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:16:06.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:06 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:06.574+0000 7f08f3818200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:16:07.160 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:06 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/2048669541' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:16:07.477 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:07 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:07.267+0000 7f08f3818200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:16:07.478 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:07 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:07.363+0000 7f08f3818200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:16:07.482 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-03-11T22:16:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:16:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "c71f9f14-dff4-11ee-95c8-87774f69a715", 2024-03-11T22:16:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T22:16:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T22:16:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T22:16:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T22:16:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T22:16:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T22:16:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-03-11T22:16:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:16:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T22:16:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T22:16:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:16:07.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-11T22:16:07.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T22:16:07.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:16:07.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T22:16:07.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T22:16:07.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:07.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T22:16:07.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:16:07.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T22:16:07.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T22:16:07.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T22:16:07.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T22:16:07.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T22:16:07.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T22:16:07.488 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:07.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T22:16:07.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T22:16:07.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T22:16:07.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T22:16:07.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T22:16:07.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T22:16:07.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T22:16:07.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T22:16:07.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T22:16:07.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:07.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T22:16:07.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:16:07.490 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T22:16:07.490 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T22:16:07.490 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:07.490 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T22:16:07.490 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T22:16:07.490 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T22:16:07.490 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T22:16:07.490 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T22:16:07.490 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T22:16:07.490 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T22:16:07.490 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:16:07.490 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:16:07.490 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:07.490 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T22:16:07.491 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:16:07.491 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T22:15:57.828352+0000", 2024-03-11T22:16:07.491 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:16:07.491 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:07.491 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T22:16:07.491 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:16:07.491 INFO:teuthology.orchestra.run.smithi027.stdout:mgr not available, waiting (1/15)... 2024-03-11T22:16:07.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:07 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:07.509+0000 7f08f3818200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:16:07.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:07 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:07.608+0000 7f08f3818200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:16:07.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:07 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:07.694+0000 7f08f3818200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:16:08.270 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:07 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:07.977+0000 7f08f3818200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:16:08.270 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:08 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:08.092+0000 7f08f3818200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:16:08.270 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:08 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:08.182+0000 7f08f3818200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:16:08.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:08 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:08.279+0000 7f08f3818200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:16:08.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:08 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:08.460+0000 7f08f3818200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:16:09.735 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:09 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:09.484+0000 7f08f3818200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:16:09.735 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:09 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:09.623+0000 7f08f3818200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:16:10.020 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:09 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:09.978+0000 7f08f3818200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:16:10.471 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:10 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/399203394' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:16:10.471 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:10 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:10.073+0000 7f08f3818200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:16:10.471 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:10 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:10.279+0000 7f08f3818200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:16:10.471 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:10 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:10.470+0000 7f08f3818200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:16:10.582 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-03-11T22:16:10.583 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:16:10.583 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "c71f9f14-dff4-11ee-95c8-87774f69a715", 2024-03-11T22:16:10.583 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T22:16:10.583 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T22:16:10.583 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T22:16:10.583 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T22:16:10.583 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:10.583 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T22:16:10.583 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T22:16:10.583 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-03-11T22:16:10.583 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:16:10.583 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T22:16:10.583 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T22:16:10.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:16:10.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-11T22:16:10.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T22:16:10.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:16:10.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T22:16:10.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T22:16:10.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:10.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T22:16:10.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:16:10.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T22:16:10.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T22:16:10.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T22:16:10.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T22:16:10.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T22:16:10.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T22:16:10.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:10.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T22:16:10.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T22:16:10.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T22:16:10.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T22:16:10.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T22:16:10.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T22:16:10.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T22:16:10.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T22:16:10.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T22:16:10.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:10.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T22:16:10.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:16:10.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T22:16:10.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T22:16:10.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:10.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T22:16:10.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T22:16:10.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T22:16:10.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T22:16:10.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T22:16:10.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T22:16:10.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T22:16:10.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:16:10.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:16:10.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:10.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T22:16:10.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:16:10.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T22:15:57.828352+0000", 2024-03-11T22:16:10.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:16:10.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:10.587 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T22:16:10.587 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:16:10.587 INFO:teuthology.orchestra.run.smithi027.stdout:mgr not available, waiting (2/15)... 2024-03-11T22:16:11.020 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:10 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:10.765+0000 7f08f3818200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:16:11.333 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:11 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:11.068+0000 7f08f3818200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:16:11.333 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:11 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:11.333+0000 7f08f3818200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:16:12.520 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:12 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:12.170+0000 7f08f3818200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:16:12.963 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:12 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:12.774+0000 7f08f3818200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:16:12.963 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:12 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:12.878+0000 7f08f3818200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:16:13.270 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:13 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:13.106+0000 7f08f3818200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:16:13.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:13 smithi027 ceph-mon[25797]: Activating manager daemon y 2024-03-11T22:16:13.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:13 smithi027 ceph-mon[25797]: mgrmap e2: y(active, starting, since 0.00357384s) 2024-03-11T22:16:13.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:13 smithi027 ceph-mon[25797]: from='mgr.14100 172.21.15.27:0/507691589' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:16:13.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:13 smithi027 ceph-mon[25797]: from='mgr.14100 172.21.15.27:0/507691589' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:16:13.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:13 smithi027 ceph-mon[25797]: from='mgr.14100 172.21.15.27:0/507691589' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:16:13.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:13 smithi027 ceph-mon[25797]: from='mgr.14100 172.21.15.27:0/507691589' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:16:13.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:13 smithi027 ceph-mon[25797]: from='mgr.14100 172.21.15.27:0/507691589' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:16:13.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:13 smithi027 ceph-mon[25797]: Manager daemon y is now available 2024-03-11T22:16:13.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:13 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/2841320137' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:16:13.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:13 smithi027 ceph-mon[25797]: from='mgr.14100 172.21.15.27:0/507691589' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:16:13.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:13 smithi027 ceph-mon[25797]: from='mgr.14100 172.21.15.27:0/507691589' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:16:13.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:13 smithi027 ceph-mon[25797]: from='mgr.14100 172.21.15.27:0/507691589' entity='mgr.y' 2024-03-11T22:16:13.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:13 smithi027 ceph-mon[25797]: from='mgr.14100 172.21.15.27:0/507691589' entity='mgr.y' 2024-03-11T22:16:13.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:13 smithi027 ceph-mon[25797]: from='mgr.14100 172.21.15.27:0/507691589' entity='mgr.y' 2024-03-11T22:16:13.704 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-03-11T22:16:13.705 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:16:13.705 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "c71f9f14-dff4-11ee-95c8-87774f69a715", 2024-03-11T22:16:13.705 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T22:16:13.705 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T22:16:13.705 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T22:16:13.705 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T22:16:13.705 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:13.705 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T22:16:13.705 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T22:16:13.705 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-03-11T22:16:13.705 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:16:13.706 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T22:16:13.706 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T22:16:13.706 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:16:13.706 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-11T22:16:13.706 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T22:16:13.706 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:16:13.706 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T22:16:13.706 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T22:16:13.706 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:13.706 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T22:16:13.706 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:16:13.706 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T22:16:13.706 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T22:16:13.707 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T22:16:13.707 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T22:16:13.707 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T22:16:13.707 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T22:16:13.710 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:13.710 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T22:16:13.710 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T22:16:13.710 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T22:16:13.710 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T22:16:13.710 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T22:16:13.710 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T22:16:13.711 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T22:16:13.711 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T22:16:13.711 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T22:16:13.711 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:13.711 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T22:16:13.711 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:16:13.711 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T22:16:13.711 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T22:16:13.711 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:13.711 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T22:16:13.711 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T22:16:13.711 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T22:16:13.711 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T22:16:13.711 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T22:16:13.711 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T22:16:13.712 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T22:16:13.712 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:16:13.712 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:16:13.712 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:13.712 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T22:16:13.712 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:16:13.712 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T22:15:57.828352+0000", 2024-03-11T22:16:13.712 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:16:13.712 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:13.712 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T22:16:13.712 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:16:13.712 INFO:teuthology.orchestra.run.smithi027.stdout:mgr not available, waiting (3/15)... 2024-03-11T22:16:15.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:15 smithi027 ceph-mon[25797]: mgrmap e3: y(active, since 1.00774s) 2024-03-11T22:16:17.078 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-03-11T22:16:17.078 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:16:17.078 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "c71f9f14-dff4-11ee-95c8-87774f69a715", 2024-03-11T22:16:17.078 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T22:16:17.078 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T22:16:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T22:16:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T22:16:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T22:16:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T22:16:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-03-11T22:16:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:16:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T22:16:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T22:16:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:16:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-11T22:16:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T22:16:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:16:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T22:16:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T22:16:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:17.080 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T22:16:17.080 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:16:17.080 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T22:16:17.080 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T22:16:17.080 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T22:16:17.080 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T22:16:17.080 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T22:16:17.080 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T22:16:17.080 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:17.080 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T22:16:17.080 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T22:16:17.084 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T22:16:17.084 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T22:16:17.084 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T22:16:17.084 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T22:16:17.084 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T22:16:17.084 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T22:16:17.084 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T22:16:17.084 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:17.084 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T22:16:17.084 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:16:17.084 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T22:16:17.084 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T22:16:17.085 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:17.085 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T22:16:17.085 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T22:16:17.085 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T22:16:17.085 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T22:16:17.085 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T22:16:17.085 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T22:16:17.085 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T22:16:17.085 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:16:17.085 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:16:17.085 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:17.085 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T22:16:17.085 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:16:17.085 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T22:15:57.828352+0000", 2024-03-11T22:16:17.085 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:16:17.086 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:16:17.086 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T22:16:17.086 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:16:17.086 INFO:teuthology.orchestra.run.smithi027.stdout:mgr is available 2024-03-11T22:16:17.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:17 smithi027 ceph-mon[25797]: mgrmap e4: y(active, since 3s) 2024-03-11T22:16:17.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:17 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/3615091795' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:16:18.320 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:18 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/839696585' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T22:16:18.320 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:18 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/839696585' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T22:16:18.324 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-03-11T22:16:18.324 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T22:16:18.324 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout fsid = c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:16:18.324 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T22:16:18.324 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T22:16:18.324 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T22:16:18.324 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T22:16:18.324 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-03-11T22:16:18.324 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T22:16:18.324 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T22:16:18.324 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-03-11T22:16:18.324 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T22:16:18.325 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T22:16:18.325 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T22:16:18.325 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling cephadm module... 2024-03-11T22:16:19.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:19 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/3224027502' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-11T22:16:19.520 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:19 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: ignoring --setuser ceph since I am not root 2024-03-11T22:16:19.520 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:19 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: ignoring --setgroup ceph since I am not root 2024-03-11T22:16:19.888 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:19 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:19.715+0000 7f3fade8b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:16:19.888 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:19 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:19.815+0000 7f3fade8b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:16:20.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:20 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/3224027502' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-11T22:16:20.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:20 smithi027 ceph-mon[25797]: mgrmap e5: y(active, since 6s) 2024-03-11T22:16:20.689 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:20 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:20.505+0000 7f3fade8b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:16:20.690 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:20 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:20.603+0000 7f3fade8b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:16:20.961 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:20 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:20.750+0000 7f3fade8b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:16:20.961 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:20 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:20.850+0000 7f3fade8b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:16:20.961 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:20 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:20.937+0000 7f3fade8b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:16:20.964 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:16:20.964 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-11T22:16:20.964 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T22:16:20.964 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T22:16:20.964 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T22:16:20.964 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:16:20.964 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for the mgr to restart... 2024-03-11T22:16:20.964 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr epoch 5... 2024-03-11T22:16:21.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:21 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/2448061078' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T22:16:21.520 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:21 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:21.222+0000 7f3fade8b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:16:21.520 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:21 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:21.341+0000 7f3fade8b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:16:21.520 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:21 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:21.432+0000 7f3fade8b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:16:22.020 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:21 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:21.532+0000 7f3fade8b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:16:22.020 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:21 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:21.714+0000 7f3fade8b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:16:23.020 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:22 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:22.732+0000 7f3fade8b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:16:23.020 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:22 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:22.869+0000 7f3fade8b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:16:23.510 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:23 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:23.214+0000 7f3fade8b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:16:23.510 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:23 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:23.308+0000 7f3fade8b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:16:23.511 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:23 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:23.511+0000 7f3fade8b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:16:23.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:23 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:23.702+0000 7f3fade8b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:16:24.267 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:23 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:23.998+0000 7f3fade8b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:16:24.267 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:24 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:24.267+0000 7f3fade8b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:16:24.520 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:24 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:24.513+0000 7f3fade8b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:16:25.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:25 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:25.348+0000 7f3fade8b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:16:26.270 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:25 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:25.944+0000 7f3fade8b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:16:26.270 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:26 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:26.046+0000 7f3fade8b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:16:26.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:26 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:26.272+0000 7f3fade8b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:16:26.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:26 smithi027 ceph-mon[25797]: Active manager daemon y restarted 2024-03-11T22:16:26.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:26 smithi027 ceph-mon[25797]: Activating manager daemon y 2024-03-11T22:16:26.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:26 smithi027 ceph-mon[25797]: osdmap e2: 0 total, 0 up, 0 in 2024-03-11T22:16:26.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:26 smithi027 ceph-mon[25797]: mgrmap e6: y(active, starting, since 0.00376209s) 2024-03-11T22:16:26.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:26 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:16:26.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:26 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:16:26.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:26 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:16:26.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:26 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:16:26.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:26 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:16:26.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:26 smithi027 ceph-mon[25797]: Manager daemon y is now available 2024-03-11T22:16:26.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:26 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' 2024-03-11T22:16:26.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:26 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' 2024-03-11T22:16:26.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:26 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:16:26.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:26 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:16:27.595 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:27 smithi027 ceph-mon[25797]: Found migration_current of "None". Setting to last migration. 2024-03-11T22:16:27.596 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:27 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:16:27.596 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:27 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:16:27.596 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:27 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:16:27.596 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:27 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' 2024-03-11T22:16:27.596 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:27 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' 2024-03-11T22:16:27.596 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:27 smithi027 ceph-mon[25797]: mgrmap e7: y(active, since 1.0054s) 2024-03-11T22:16:27.849 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:16:27.849 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-11T22:16:27.849 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T22:16:27.849 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:16:27.849 INFO:teuthology.orchestra.run.smithi027.stdout:mgr epoch 5 is available 2024-03-11T22:16:27.849 INFO:teuthology.orchestra.run.smithi027.stdout:Setting orchestrator backend to cephadm... 2024-03-11T22:16:28.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:28 smithi027 ceph-mon[25797]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T22:16:28.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:28 smithi027 ceph-mon[25797]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T22:16:29.420 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:29 smithi027 ceph-mon[25797]: mgrmap e8: y(active, since 2s) 2024-03-11T22:16:29.421 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:29 smithi027 ceph-mon[25797]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:16:29.421 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:29 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' 2024-03-11T22:16:29.421 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:29 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:16:30.319 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-11T22:16:30.319 INFO:teuthology.orchestra.run.smithi027.stdout:Generating ssh key... 2024-03-11T22:16:30.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:30 smithi027 ceph-mon[25797]: [11/Mar/2024:22:16:29] ENGINE Bus STARTING 2024-03-11T22:16:30.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:30 smithi027 ceph-mon[25797]: [11/Mar/2024:22:16:29] ENGINE Serving on http://172.21.15.27:8765 2024-03-11T22:16:30.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:30 smithi027 ceph-mon[25797]: [11/Mar/2024:22:16:29] ENGINE Serving on https://172.21.15.27:7150 2024-03-11T22:16:30.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:30 smithi027 ceph-mon[25797]: [11/Mar/2024:22:16:29] ENGINE Bus STARTED 2024-03-11T22:16:30.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:30 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:16:30.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:30 smithi027 ceph-mon[25797]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:16:31.473 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:31 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: Generating public/private rsa key pair. 2024-03-11T22:16:31.473 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:31 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: Your identification has been saved in /tmp/tmpgbaap7nf/key. 2024-03-11T22:16:31.473 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:31 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: Your public key has been saved in /tmp/tmpgbaap7nf/key.pub. 2024-03-11T22:16:31.473 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:31 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: The key fingerprint is: 2024-03-11T22:16:31.473 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:31 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: SHA256:7HXF5n8atMxxFXX4X+iULW6R9B+w8UBoGyKlkR/yirU ceph-c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:16:31.473 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:31 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: The key's randomart image is: 2024-03-11T22:16:31.474 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:31 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: +---[RSA 3072]----+ 2024-03-11T22:16:31.474 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:31 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: | .o. o. o+| 2024-03-11T22:16:31.474 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:31 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: | +oo + =o o| 2024-03-11T22:16:31.474 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:31 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: | .= + o.XB.| 2024-03-11T22:16:31.474 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:31 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: | o o . =Bo*| 2024-03-11T22:16:31.474 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:31 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: | o S . .++o*| 2024-03-11T22:16:31.474 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:31 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: | . E . . ++=o| 2024-03-11T22:16:31.474 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:31 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: | . .= o| 2024-03-11T22:16:31.474 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:31 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: | o.| 2024-03-11T22:16:31.474 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:31 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: | . | 2024-03-11T22:16:31.474 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:31 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: +----[SHA256]-----+ 2024-03-11T22:16:32.428 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:32 smithi027 ceph-mon[25797]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:16:32.429 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:32 smithi027 ceph-mon[25797]: Generating ssh key... 2024-03-11T22:16:32.429 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:32 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' 2024-03-11T22:16:32.429 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:32 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' 2024-03-11T22:16:32.947 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQalKRQblX4kZ799t+nUT2N71aWVaQOkwbQDJY+8eLnUcLdlVlSWBUSm8a7J0Km7Y0y1zNwHKHWs7Mho8WLfQ4A9NJULy260RXrfrmUhzL+I7N7Ga2MgSmmwlg5eq1A1HAhyzCnujKxLanJw8vjj5ZgV1IxWRvqkmFUjkcLhXfiOxA6Z5vMXxCXDLc1W2+NRMdJS8s/D7sMk1dtApRmaYTuI+aWlv7Ra/9iJdguIqN5cf1wL2ZwaTrCsmHQdUXKBIbM0qla9RKvVgwfQGkUGQboxDQSLFcVLPNAPRpo5/7v/DY0Rmm3JSB8680Fjy0YKZn+LOeNE0Z1Aa9VcXoTQi/oLGPHrwFaJyU3VBKE0y2vMI464JsBG7awYXT2+pmcktXxUQoqfbpBGNIabfI1JtbgGsGTrg10LQUwBD1ANTp0Moyge7U84Aem366sJz4tQM1+wC5mIpd0qJOPvQQnorhKpUoVobnR70v1OSELvbHY1Bwf8vkC04XYgcweu1a1Ek= ceph-c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:16:32.947 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:16:32.947 INFO:teuthology.orchestra.run.smithi027.stdout:Adding key to root@localhost authorized_keys... 2024-03-11T22:16:32.948 INFO:teuthology.orchestra.run.smithi027.stdout:Adding host smithi027... 2024-03-11T22:16:33.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:33 smithi027 ceph-mon[25797]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:16:34.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:34 smithi027 ceph-mon[25797]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi027", "addr": "172.21.15.27", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:16:35.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:35 smithi027 ceph-mon[25797]: Deploying cephadm binary to smithi027 2024-03-11T22:16:38.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Added host 'smithi027' with addr '172.21.15.27' 2024-03-11T22:16:38.697 INFO:teuthology.orchestra.run.smithi027.stdout:Deploying unmanaged mon service... 2024-03-11T22:16:39.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:39 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' 2024-03-11T22:16:39.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:39 smithi027 ceph-mon[25797]: Added host smithi027 2024-03-11T22:16:39.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:39 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:16:39.962 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-11T22:16:39.963 INFO:teuthology.orchestra.run.smithi027.stdout:Deploying unmanaged mgr service... 2024-03-11T22:16:40.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:40 smithi027 ceph-mon[25797]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:16:40.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:40 smithi027 ceph-mon[25797]: Saving service mon spec with placement count:5 2024-03-11T22:16:40.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:40 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' 2024-03-11T22:16:42.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:42 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' 2024-03-11T22:16:42.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:42 smithi027 ceph-mon[25797]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:16:42.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:42 smithi027 ceph-mon[25797]: Saving service mgr spec with placement count:2 2024-03-11T22:16:42.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:42 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' 2024-03-11T22:16:42.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:42 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' 2024-03-11T22:16:42.341 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-11T22:16:44.115 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:44 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/958335100' entity='client.admin' 2024-03-11T22:16:45.498 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling the dashboard module... 2024-03-11T22:16:45.749 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:45 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' 2024-03-11T22:16:45.750 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:45 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/236442899' entity='client.admin' 2024-03-11T22:16:47.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:47 smithi027 ceph-mon[25797]: from='mgr.14120 172.21.15.27:0/1583195406' entity='mgr.y' 2024-03-11T22:16:47.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:47 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/1815349195' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-11T22:16:47.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:47 smithi027 ceph-mon[25797]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:16:47.521 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:47 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: ignoring --setuser ceph since I am not root 2024-03-11T22:16:47.521 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:47 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: ignoring --setgroup ceph since I am not root 2024-03-11T22:16:47.847 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:47 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:47.813+0000 7f45688a2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:16:48.227 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:47 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:47.912+0000 7f45688a2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:16:48.498 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:48 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/1815349195' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-11T22:16:48.498 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:48 smithi027 ceph-mon[25797]: mgrmap e9: y(active, since 20s) 2024-03-11T22:16:48.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:48 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:48.593+0000 7f45688a2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:16:48.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:48 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:48.692+0000 7f45688a2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:16:49.228 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:48 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:48.842+0000 7f45688a2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:16:49.228 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:48 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:48.941+0000 7f45688a2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:16:49.228 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:49 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:49.035+0000 7f45688a2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:16:49.518 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:49 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/480832193' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T22:16:49.518 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:49 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:49.312+0000 7f45688a2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:16:49.518 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:49 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:49.427+0000 7f45688a2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:16:49.518 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:49 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:49.517+0000 7f45688a2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:16:49.643 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:16:49.643 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-11T22:16:49.643 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T22:16:49.643 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T22:16:49.644 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T22:16:49.644 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:16:49.644 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for the mgr to restart... 2024-03-11T22:16:49.644 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr epoch 9... 2024-03-11T22:16:49.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:49 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:49.616+0000 7f45688a2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:16:50.066 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:49 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:49.803+0000 7f45688a2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:16:51.270 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:50.829+0000 7f45688a2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:16:51.270 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:50.965+0000 7f45688a2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:16:51.610 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:51 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:51.311+0000 7f45688a2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:16:51.610 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:51 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:51.405+0000 7f45688a2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:16:51.610 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:51 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:51.609+0000 7f45688a2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:16:52.020 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:51 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:51.801+0000 7f45688a2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:16:52.367 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:52 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:52.096+0000 7f45688a2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:16:52.368 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:52 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:52.364+0000 7f45688a2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:16:52.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:52 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:52.610+0000 7f45688a2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:16:53.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:53 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:53.449+0000 7f45688a2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:16:54.374 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:54 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:54.045+0000 7f45688a2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:16:54.374 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:54 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:54.147+0000 7f45688a2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:16:54.375 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:16:54 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:16:54.373+0000 7f45688a2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:16:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:54 smithi027 ceph-mon[25797]: Active manager daemon y restarted 2024-03-11T22:16:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:54 smithi027 ceph-mon[25797]: Activating manager daemon y 2024-03-11T22:16:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:54 smithi027 ceph-mon[25797]: osdmap e3: 0 total, 0 up, 0 in 2024-03-11T22:16:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:54 smithi027 ceph-mon[25797]: mgrmap e10: y(active, starting, since 0.00333167s) 2024-03-11T22:16:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:54 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:16:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:54 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:16:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:54 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:16:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:54 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:16:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:54 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:16:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:54 smithi027 ceph-mon[25797]: Manager daemon y is now available 2024-03-11T22:16:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:54 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:16:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:54 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:16:55.751 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:55 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:16:55.752 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:55 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:16:55.752 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:55 smithi027 ceph-mon[25797]: [11/Mar/2024:22:16:54] ENGINE Bus STARTING 2024-03-11T22:16:55.752 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:55 smithi027 ceph-mon[25797]: [11/Mar/2024:22:16:54] ENGINE Serving on http://172.21.15.27:8765 2024-03-11T22:16:55.752 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:55 smithi027 ceph-mon[25797]: [11/Mar/2024:22:16:55] ENGINE Serving on https://172.21.15.27:7150 2024-03-11T22:16:55.752 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:55 smithi027 ceph-mon[25797]: [11/Mar/2024:22:16:55] ENGINE Bus STARTED 2024-03-11T22:16:55.753 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:55 smithi027 ceph-mon[25797]: mgrmap e11: y(active, since 1.0058s) 2024-03-11T22:16:56.259 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:16:56.260 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-11T22:16:56.260 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T22:16:56.260 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:16:56.260 INFO:teuthology.orchestra.run.smithi027.stdout:mgr epoch 9 is available 2024-03-11T22:16:56.260 INFO:teuthology.orchestra.run.smithi027.stdout:Generating a dashboard self-signed certificate... 2024-03-11T22:16:56.626 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:56 smithi027 ceph-mon[25797]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T22:16:56.626 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:56 smithi027 ceph-mon[25797]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T22:16:57.646 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:57 smithi027 ceph-mon[25797]: mgrmap e12: y(active, since 2s) 2024-03-11T22:16:57.646 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:16:57.646 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:57 smithi027 ceph-mon[25797]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:16:57.646 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:16:57.646 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:16:57.648 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-11T22:16:57.648 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial admin user... 2024-03-11T22:16:59.012 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:58 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:16:59.012 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:58 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:16:59.866 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:16:59 smithi027 ceph-mon[25797]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:16:59.869 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$0ng5Puk6gVpYXa4bi5DMsOT2p0Hb/2yh0BN37VilAV5zGEKRetBBS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710195418, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-11T22:16:59.869 INFO:teuthology.orchestra.run.smithi027.stdout:Fetching dashboard port number... 2024-03-11T22:17:00.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:00 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/157550880' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-11T22:17:01.124 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 8443 2024-03-11T22:17:01.134 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T22:17:01.134 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-03-11T22:17:01.140 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T22:17:01.141 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-03-11T22:17:01.141 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-03-11T22:17:01.141 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-11T22:17:01.142 INFO:teuthology.orchestra.run.smithi027.stdout:Ceph Dashboard is now available at: 2024-03-11T22:17:01.142 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:17:01.142 INFO:teuthology.orchestra.run.smithi027.stdout: URL: https://smithi027.front.sepia.ceph.com:8443/ 2024-03-11T22:17:01.142 INFO:teuthology.orchestra.run.smithi027.stdout: User: admin 2024-03-11T22:17:01.143 INFO:teuthology.orchestra.run.smithi027.stdout: Password: rthh0wadwa 2024-03-11T22:17:01.143 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:17:01.143 INFO:teuthology.orchestra.run.smithi027.stdout:Saving cluster configuration to /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config directory 2024-03-11T22:17:01.143 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling autotune for osd_memory_target 2024-03-11T22:17:02.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:01 smithi027 ceph-mon[25797]: mgrmap e13: y(active, since 6s) 2024-03-11T22:17:03.770 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-11T22:17:03.770 INFO:teuthology.orchestra.run.smithi027.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-11T22:17:03.770 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:17:03.770 INFO:teuthology.orchestra.run.smithi027.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:17:03.770 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:17:03.770 INFO:teuthology.orchestra.run.smithi027.stdout:Or, if you are only running a single cluster on this host: 2024-03-11T22:17:03.770 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:17:03.770 INFO:teuthology.orchestra.run.smithi027.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-11T22:17:03.770 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:17:03.771 INFO:teuthology.orchestra.run.smithi027.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-11T22:17:03.771 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:17:03.771 INFO:teuthology.orchestra.run.smithi027.stdout: ceph telemetry on 2024-03-11T22:17:03.771 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:17:03.771 INFO:teuthology.orchestra.run.smithi027.stdout:For more information see: 2024-03-11T22:17:03.771 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:17:03.771 INFO:teuthology.orchestra.run.smithi027.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-11T22:17:03.771 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:17:03.771 INFO:teuthology.orchestra.run.smithi027.stdout:Bootstrap complete. 2024-03-11T22:17:03.814 INFO:tasks.cephadm:Fetching config... 2024-03-11T22:17:03.815 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-11T22:17:03.815 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-11T22:17:03.831 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-11T22:17:03.831 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-11T22:17:03.831 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-11T22:17:03.887 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-11T22:17:03.887 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-11T22:17:03.887 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-11T22:17:03.959 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-11T22:17:03.959 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-11T22:17:03.959 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-11T22:17:04.013 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-11T22:17:04.014 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQalKRQblX4kZ799t+nUT2N71aWVaQOkwbQDJY+8eLnUcLdlVlSWBUSm8a7J0Km7Y0y1zNwHKHWs7Mho8WLfQ4A9NJULy260RXrfrmUhzL+I7N7Ga2MgSmmwlg5eq1A1HAhyzCnujKxLanJw8vjj5ZgV1IxWRvqkmFUjkcLhXfiOxA6Z5vMXxCXDLc1W2+NRMdJS8s/D7sMk1dtApRmaYTuI+aWlv7Ra/9iJdguIqN5cf1wL2ZwaTrCsmHQdUXKBIbM0qla9RKvVgwfQGkUGQboxDQSLFcVLPNAPRpo5/7v/DY0Rmm3JSB8680Fjy0YKZn+LOeNE0Z1Aa9VcXoTQi/oLGPHrwFaJyU3VBKE0y2vMI464JsBG7awYXT2+pmcktXxUQoqfbpBGNIabfI1JtbgGsGTrg10LQUwBD1ANTp0Moyge7U84Aem366sJz4tQM1+wC5mIpd0qJOPvQQnorhKpUoVobnR70v1OSELvbHY1Bwf8vkC04XYgcweu1a1Ek= ceph-c71f9f14-dff4-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T22:17:04.099 INFO:teuthology.orchestra.run.smithi027.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQalKRQblX4kZ799t+nUT2N71aWVaQOkwbQDJY+8eLnUcLdlVlSWBUSm8a7J0Km7Y0y1zNwHKHWs7Mho8WLfQ4A9NJULy260RXrfrmUhzL+I7N7Ga2MgSmmwlg5eq1A1HAhyzCnujKxLanJw8vjj5ZgV1IxWRvqkmFUjkcLhXfiOxA6Z5vMXxCXDLc1W2+NRMdJS8s/D7sMk1dtApRmaYTuI+aWlv7Ra/9iJdguIqN5cf1wL2ZwaTrCsmHQdUXKBIbM0qla9RKvVgwfQGkUGQboxDQSLFcVLPNAPRpo5/7v/DY0Rmm3JSB8680Fjy0YKZn+LOeNE0Z1Aa9VcXoTQi/oLGPHrwFaJyU3VBKE0y2vMI464JsBG7awYXT2+pmcktXxUQoqfbpBGNIabfI1JtbgGsGTrg10LQUwBD1ANTp0Moyge7U84Aem366sJz4tQM1+wC5mIpd0qJOPvQQnorhKpUoVobnR70v1OSELvbHY1Bwf8vkC04XYgcweu1a1Ek= ceph-c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:17:04.123 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQalKRQblX4kZ799t+nUT2N71aWVaQOkwbQDJY+8eLnUcLdlVlSWBUSm8a7J0Km7Y0y1zNwHKHWs7Mho8WLfQ4A9NJULy260RXrfrmUhzL+I7N7Ga2MgSmmwlg5eq1A1HAhyzCnujKxLanJw8vjj5ZgV1IxWRvqkmFUjkcLhXfiOxA6Z5vMXxCXDLc1W2+NRMdJS8s/D7sMk1dtApRmaYTuI+aWlv7Ra/9iJdguIqN5cf1wL2ZwaTrCsmHQdUXKBIbM0qla9RKvVgwfQGkUGQboxDQSLFcVLPNAPRpo5/7v/DY0Rmm3JSB8680Fjy0YKZn+LOeNE0Z1Aa9VcXoTQi/oLGPHrwFaJyU3VBKE0y2vMI464JsBG7awYXT2+pmcktXxUQoqfbpBGNIabfI1JtbgGsGTrg10LQUwBD1ANTp0Moyge7U84Aem366sJz4tQM1+wC5mIpd0qJOPvQQnorhKpUoVobnR70v1OSELvbHY1Bwf8vkC04XYgcweu1a1Ek= ceph-c71f9f14-dff4-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T22:17:04.169 INFO:teuthology.orchestra.run.smithi174.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQalKRQblX4kZ799t+nUT2N71aWVaQOkwbQDJY+8eLnUcLdlVlSWBUSm8a7J0Km7Y0y1zNwHKHWs7Mho8WLfQ4A9NJULy260RXrfrmUhzL+I7N7Ga2MgSmmwlg5eq1A1HAhyzCnujKxLanJw8vjj5ZgV1IxWRvqkmFUjkcLhXfiOxA6Z5vMXxCXDLc1W2+NRMdJS8s/D7sMk1dtApRmaYTuI+aWlv7Ra/9iJdguIqN5cf1wL2ZwaTrCsmHQdUXKBIbM0qla9RKvVgwfQGkUGQboxDQSLFcVLPNAPRpo5/7v/DY0Rmm3JSB8680Fjy0YKZn+LOeNE0Z1Aa9VcXoTQi/oLGPHrwFaJyU3VBKE0y2vMI464JsBG7awYXT2+pmcktXxUQoqfbpBGNIabfI1JtbgGsGTrg10LQUwBD1ANTp0Moyge7U84Aem366sJz4tQM1+wC5mIpd0qJOPvQQnorhKpUoVobnR70v1OSELvbHY1Bwf8vkC04XYgcweu1a1Ek= ceph-c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:17:04.184 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQalKRQblX4kZ799t+nUT2N71aWVaQOkwbQDJY+8eLnUcLdlVlSWBUSm8a7J0Km7Y0y1zNwHKHWs7Mho8WLfQ4A9NJULy260RXrfrmUhzL+I7N7Ga2MgSmmwlg5eq1A1HAhyzCnujKxLanJw8vjj5ZgV1IxWRvqkmFUjkcLhXfiOxA6Z5vMXxCXDLc1W2+NRMdJS8s/D7sMk1dtApRmaYTuI+aWlv7Ra/9iJdguIqN5cf1wL2ZwaTrCsmHQdUXKBIbM0qla9RKvVgwfQGkUGQboxDQSLFcVLPNAPRpo5/7v/DY0Rmm3JSB8680Fjy0YKZn+LOeNE0Z1Aa9VcXoTQi/oLGPHrwFaJyU3VBKE0y2vMI464JsBG7awYXT2+pmcktXxUQoqfbpBGNIabfI1JtbgGsGTrg10LQUwBD1ANTp0Moyge7U84Aem366sJz4tQM1+wC5mIpd0qJOPvQQnorhKpUoVobnR70v1OSELvbHY1Bwf8vkC04XYgcweu1a1Ek= ceph-c71f9f14-dff4-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T22:17:04.229 INFO:teuthology.orchestra.run.smithi183.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQalKRQblX4kZ799t+nUT2N71aWVaQOkwbQDJY+8eLnUcLdlVlSWBUSm8a7J0Km7Y0y1zNwHKHWs7Mho8WLfQ4A9NJULy260RXrfrmUhzL+I7N7Ga2MgSmmwlg5eq1A1HAhyzCnujKxLanJw8vjj5ZgV1IxWRvqkmFUjkcLhXfiOxA6Z5vMXxCXDLc1W2+NRMdJS8s/D7sMk1dtApRmaYTuI+aWlv7Ra/9iJdguIqN5cf1wL2ZwaTrCsmHQdUXKBIbM0qla9RKvVgwfQGkUGQboxDQSLFcVLPNAPRpo5/7v/DY0Rmm3JSB8680Fjy0YKZn+LOeNE0Z1Aa9VcXoTQi/oLGPHrwFaJyU3VBKE0y2vMI464JsBG7awYXT2+pmcktXxUQoqfbpBGNIabfI1JtbgGsGTrg10LQUwBD1ANTp0Moyge7U84Aem366sJz4tQM1+wC5mIpd0qJOPvQQnorhKpUoVobnR70v1OSELvbHY1Bwf8vkC04XYgcweu1a1Ek= ceph-c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:17:04.245 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-11T22:17:04.379 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:04 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/3550591737' entity='client.admin' 2024-03-11T22:17:04.500 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:17:07.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:07 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:07.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:07 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:07.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:07 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi027", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:17:07.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:07 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:07.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:07 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:07.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:07 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:07.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:07 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/3298195113' entity='client.admin' 2024-03-11T22:17:07.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:07 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:07.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:07 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:07.843 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-11T22:17:07.844 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-11T22:17:08.631 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:17:11.013 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi174 2024-03-11T22:17:11.014 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-11T22:17:11.014 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/etc/ceph/ceph.conf 2024-03-11T22:17:11.032 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-11T22:17:11.032 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:17:11.090 INFO:tasks.cephadm:Adding host smithi174 to orchestrator... 2024-03-11T22:17:11.090 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph orch host add smithi174 2024-03-11T22:17:11.339 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:17:11.564 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:11 smithi027 ceph-mon[25797]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:11.564 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:11 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:11.565 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:11 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:11.565 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:11 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:11.565 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:11 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:17:11.565 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:11 smithi027 ceph-mon[25797]: Updating smithi027:/etc/ceph/ceph.conf 2024-03-11T22:17:11.565 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:11 smithi027 ceph-mon[25797]: Updating smithi027:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:17:11.565 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:11 smithi027 ceph-mon[25797]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:17:11.565 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:11 smithi027 ceph-mon[25797]: Updating smithi027:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:17:11.565 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:11 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:11.565 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:11 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:11.565 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:11 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:13.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:13 smithi027 ceph-mon[25797]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi174", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:14.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:14 smithi027 ceph-mon[25797]: Deploying cephadm binary to smithi174 2024-03-11T22:17:15.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:15 smithi027 ceph-mon[25797]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:17.447 INFO:teuthology.orchestra.run.smithi027.stdout:Added host 'smithi174' with addr '172.21.15.174' 2024-03-11T22:17:17.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:17 smithi027 ceph-mon[25797]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:18.109 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T22:17:18.354 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:17:18.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:18 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:18.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:18 smithi027 ceph-mon[25797]: Added host smithi174 2024-03-11T22:17:18.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:18 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:18.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:18 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:19.763 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:19 smithi027 ceph-mon[25797]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:19.763 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:19 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:20.014 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:17:20.014 INFO:teuthology.orchestra.run.smithi027.stdout:[{"addr": "172.21.15.27", "hostname": "smithi027", "labels": [], "status": ""}, {"addr": "172.21.15.174", "hostname": "smithi174", "labels": [], "status": ""}] 2024-03-11T22:17:20.678 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi183 2024-03-11T22:17:20.679 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-11T22:17:20.679 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/etc/ceph/ceph.conf 2024-03-11T22:17:20.698 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-11T22:17:20.698 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:17:20.756 INFO:tasks.cephadm:Adding host smithi183 to orchestrator... 2024-03-11T22:17:20.756 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph orch host add smithi183 2024-03-11T22:17:21.000 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:17:21.344 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:21 smithi027 ceph-mon[25797]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:17:21.344 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:21 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:22.314 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:22 smithi027 ceph-mon[25797]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:24.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:24 smithi027 ceph-mon[25797]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:24.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:24 smithi027 ceph-mon[25797]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi183", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:24.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:24 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:24.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:24 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:24.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:24 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:24.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:24 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:24.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:24 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:17:24.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:24 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:24.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:24 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:17:25.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:25 smithi027 ceph-mon[25797]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-11T22:17:25.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:25 smithi027 ceph-mon[25797]: Deploying cephadm binary to smithi183 2024-03-11T22:17:25.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:25 smithi027 ceph-mon[25797]: Updating smithi174:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:17:25.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:25 smithi027 ceph-mon[25797]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:17:25.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:25 smithi027 ceph-mon[25797]: Updating smithi174:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:17:26.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:26 smithi027 ceph-mon[25797]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:27.184 INFO:teuthology.orchestra.run.smithi027.stdout:Added host 'smithi183' with addr '172.21.15.183' 2024-03-11T22:17:27.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:27 smithi027 ceph-mon[25797]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:27.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:27 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:27.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:27 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:27.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:27 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:27.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:27 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:27.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:27 smithi027 ceph-mon[25797]: Added host smithi183 2024-03-11T22:17:27.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:27 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:27.963 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T22:17:28.211 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:17:29.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:28 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:29.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:28 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:30.113 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:17:30.114 INFO:teuthology.orchestra.run.smithi027.stdout:[{"addr": "172.21.15.27", "hostname": "smithi027", "labels": [], "status": ""}, {"addr": "172.21.15.174", "hostname": "smithi174", "labels": [], "status": ""}, {"addr": "172.21.15.183", "hostname": "smithi183", "labels": [], "status": ""}] 2024-03-11T22:17:30.415 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:30 smithi027 ceph-mon[25797]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:30.415 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:30 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:30.833 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-11T22:17:30.833 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-11T22:17:31.087 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:17:31.459 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:31 smithi027 ceph-mon[25797]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:17:32.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:32 smithi027 ceph-mon[25797]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:33.228 INFO:teuthology.orchestra.run.smithi027.stderr:adjusted tunables profile to default 2024-03-11T22:17:33.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:33 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/2342907253' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T22:17:33.848 INFO:tasks.cephadm:Adding mon.a on smithi027 2024-03-11T22:17:33.848 INFO:tasks.cephadm:Adding mon.b on smithi174 2024-03-11T22:17:33.848 INFO:tasks.cephadm:Adding mon.c on smithi183 2024-03-11T22:17:33.848 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi027:172.21.15.27=a;smithi174:172.21.15.174=b;smithi183:172.21.15.183=c' 2024-03-11T22:17:34.080 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:17:34.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:34 smithi027 ceph-mon[25797]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:34.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:34 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/2342907253' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T22:17:34.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:34 smithi027 ceph-mon[25797]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:17:34.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:34 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:34.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:34 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:34.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:34 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:34.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:34 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:34.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:34 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:17:34.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:34 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:34.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:34 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:17:34.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:34 smithi027 ceph-mon[25797]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-11T22:17:34.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:34 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:34.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:34 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:34.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:34 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:35.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:35 smithi027 ceph-mon[25797]: Updating smithi183:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:17:35.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:35 smithi027 ceph-mon[25797]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:17:35.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:35 smithi027 ceph-mon[25797]: Updating smithi183:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:17:35.818 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled mon update... 2024-03-11T22:17:36.332 DEBUG:teuthology.orchestra.run.smithi174:mon.b> sudo journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mon.b.service 2024-03-11T22:17:36.335 DEBUG:teuthology.orchestra.run.smithi183:mon.c> sudo journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mon.c.service 2024-03-11T22:17:36.337 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T22:17:36.337 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:17:36.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:36 smithi027 ceph-mon[25797]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:36.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:36 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:36.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:36 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:36.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:36 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:36.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:36 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:17:36.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:36 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:36.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:36 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:17:36.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:36 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:37.092 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:17:37.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:37 smithi027 ceph-mon[25797]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi174:172.21.15.174=b;smithi183:172.21.15.183=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:37.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:37 smithi027 ceph-mon[25797]: Saving service mon spec with placement smithi027:172.21.15.27=a;smithi174:172.21.15.174=b;smithi183:172.21.15.183=c;count:3 2024-03-11T22:17:37.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:37 smithi027 ceph-mon[25797]: Deploying daemon mon.c on smithi183 2024-03-11T22:17:38.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:38 smithi027 ceph-mon[25797]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:40.412 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-11T22:17:40.412 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":1,"fsid":"c71f9f14-dff4-11ee-95c8-87774f69a715","modified":"2024-03-11T22:15:53.875211Z","created":"2024-03-11T22:15:53.875211Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:17:40.413 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 1 2024-03-11T22:17:40.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:40 smithi027 ceph-mon[25797]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:41.228 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 podman[30868]: 2024-03-11 22:17:41.120493957 +0000 UTC m=+0.132254169 container create b32fe2121d6f65c953cfe2785f69edf1b058faca64fd7c20c50876fc74fd1ae3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-c, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=squid-94a5397) 2024-03-11T22:17:41.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:41 smithi027 ceph-mon[25797]: from='client.? 172.21.15.183:0/1562450803' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:17:41.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 podman[30868]: 2024-03-11 22:17:41.314342286 +0000 UTC m=+0.326102450 container init b32fe2121d6f65c953cfe2785f69edf1b058faca64fd7c20c50876fc74fd1ae3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-11T22:17:41.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 podman[30868]: 2024-03-11 22:17:41.319439024 +0000 UTC m=+0.331199188 container start b32fe2121d6f65c953cfe2785f69edf1b058faca64fd7c20c50876fc74fd1ae3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397) 2024-03-11T22:17:41.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T22:17:41.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 7 2024-03-11T22:17:41.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: pidfile_write: ignore empty --pid-file 2024-03-11T22:17:41.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: load: jerasure load: lrc 2024-03-11T22:17:41.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: RocksDB version: 7.9.2 2024-03-11T22:17:41.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Git sha 0 2024-03-11T22:17:41.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T22:17:41.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: DB SUMMARY 2024-03-11T22:17:41.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: DB Session ID: PI1FTYS64BH3XB6ZLAL8 2024-03-11T22:17:41.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: CURRENT file: CURRENT 2024-03-11T22:17:41.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T22:17:41.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-11T22:17:41.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-11T22:17:41.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-11T22:17:41.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.error_if_exists: 0 2024-03-11T22:17:41.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.create_if_missing: 0 2024-03-11T22:17:41.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.paranoid_checks: 1 2024-03-11T22:17:41.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T22:17:41.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T22:17:41.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T22:17:41.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.env: 0x556ac588bae0 2024-03-11T22:17:41.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T22:17:41.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.info_log: 0x556ac70878c0 2024-03-11T22:17:41.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T22:17:41.607 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.statistics: (nil) 2024-03-11T22:17:41.607 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.use_fsync: 0 2024-03-11T22:17:41.607 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_log_file_size: 0 2024-03-11T22:17:41.607 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T22:17:41.607 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T22:17:41.607 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T22:17:41.607 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T22:17:41.607 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.allow_fallocate: 1 2024-03-11T22:17:41.608 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T22:17:41.608 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T22:17:41.608 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.use_direct_reads: 0 2024-03-11T22:17:41.608 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T22:17:41.608 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T22:17:41.608 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.db_log_dir: 2024-03-11T22:17:41.608 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.wal_dir: 2024-03-11T22:17:41.608 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T22:17:41.608 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T22:17:41.608 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T22:17:41.608 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T22:17:41.608 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T22:17:41.608 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T22:17:41.608 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T22:17:41.609 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T22:17:41.609 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.write_buffer_manager: 0x556ac705b400 2024-03-11T22:17:41.609 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T22:17:41.609 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T22:17:41.609 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T22:17:41.609 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T22:17:41.609 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T22:17:41.609 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T22:17:41.609 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T22:17:41.609 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T22:17:41.609 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.unordered_write: 0 2024-03-11T22:17:41.609 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T22:17:41.609 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T22:17:41.609 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T22:17:41.610 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T22:17:41.610 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.row_cache: None 2024-03-11T22:17:41.610 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.wal_filter: None 2024-03-11T22:17:41.610 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T22:17:41.610 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T22:17:41.610 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.two_write_queues: 0 2024-03-11T22:17:41.610 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T22:17:41.610 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.wal_compression: 0 2024-03-11T22:17:41.610 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.atomic_flush: 0 2024-03-11T22:17:41.610 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T22:17:41.610 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T22:17:41.610 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T22:17:41.610 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.log_readahead_size: 0 2024-03-11T22:17:41.610 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T22:17:41.611 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T22:17:41.611 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T22:17:41.611 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T22:17:41.611 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T22:17:41.611 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T22:17:41.611 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T22:17:41.611 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_background_jobs: 2 2024-03-11T22:17:41.611 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_background_compactions: -1 2024-03-11T22:17:41.611 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_subcompactions: 1 2024-03-11T22:17:41.611 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T22:17:41.611 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T22:17:41.611 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T22:17:41.611 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T22:17:41.611 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T22:17:41.611 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T22:17:41.611 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T22:17:41.612 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T22:17:41.612 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_open_files: -1 2024-03-11T22:17:41.612 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T22:17:41.612 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T22:17:41.612 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T22:17:41.612 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T22:17:41.612 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_background_flushes: -1 2024-03-11T22:17:41.612 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Compression algorithms supported: 2024-03-11T22:17:41.612 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T22:17:41.612 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: kZSTD supported: 0 2024-03-11T22:17:41.612 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: kXpressCompression supported: 0 2024-03-11T22:17:41.612 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T22:17:41.612 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: kZlibCompression supported: 1 2024-03-11T22:17:41.613 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: kSnappyCompression supported: 1 2024-03-11T22:17:41.613 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: kLZ4Compression supported: 1 2024-03-11T22:17:41.613 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: kBZip2Compression supported: 0 2024-03-11T22:17:41.613 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T22:17:41.613 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T22:17:41.613 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-11T22:17:41.613 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T22:17:41.613 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T22:17:41.613 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.merge_operator: 2024-03-11T22:17:41.613 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compaction_filter: None 2024-03-11T22:17:41.613 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T22:17:41.613 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T22:17:41.613 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T22:17:41.613 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T22:17:41.614 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556ac70879c0) 2024-03-11T22:17:41.614 INFO:journalctl@ceph.mon.c.smithi183.stdout: cache_index_and_filter_blocks: 1 2024-03-11T22:17:41.614 INFO:journalctl@ceph.mon.c.smithi183.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T22:17:41.614 INFO:journalctl@ceph.mon.c.smithi183.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T22:17:41.614 INFO:journalctl@ceph.mon.c.smithi183.stdout: pin_top_level_index_and_filter: 1 2024-03-11T22:17:41.614 INFO:journalctl@ceph.mon.c.smithi183.stdout: index_type: 0 2024-03-11T22:17:41.614 INFO:journalctl@ceph.mon.c.smithi183.stdout: data_block_index_type: 0 2024-03-11T22:17:41.614 INFO:journalctl@ceph.mon.c.smithi183.stdout: index_shortening: 1 2024-03-11T22:17:41.614 INFO:journalctl@ceph.mon.c.smithi183.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T22:17:41.614 INFO:journalctl@ceph.mon.c.smithi183.stdout: checksum: 4 2024-03-11T22:17:41.614 INFO:journalctl@ceph.mon.c.smithi183.stdout: no_block_cache: 0 2024-03-11T22:17:41.614 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_cache: 0x556ac708b090 2024-03-11T22:17:41.614 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_cache_name: BinnedLRUCache 2024-03-11T22:17:41.614 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_cache_options: 2024-03-11T22:17:41.614 INFO:journalctl@ceph.mon.c.smithi183.stdout: capacity : 536870912 2024-03-11T22:17:41.615 INFO:journalctl@ceph.mon.c.smithi183.stdout: num_shard_bits : 4 2024-03-11T22:17:41.615 INFO:journalctl@ceph.mon.c.smithi183.stdout: strict_capacity_limit : 0 2024-03-11T22:17:41.615 INFO:journalctl@ceph.mon.c.smithi183.stdout: high_pri_pool_ratio: 0.000 2024-03-11T22:17:41.615 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_cache_compressed: (nil) 2024-03-11T22:17:41.615 INFO:journalctl@ceph.mon.c.smithi183.stdout: persistent_cache: (nil) 2024-03-11T22:17:41.615 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_size: 4096 2024-03-11T22:17:41.615 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_size_deviation: 10 2024-03-11T22:17:41.615 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_restart_interval: 16 2024-03-11T22:17:41.615 INFO:journalctl@ceph.mon.c.smithi183.stdout: index_block_restart_interval: 1 2024-03-11T22:17:41.615 INFO:journalctl@ceph.mon.c.smithi183.stdout: metadata_block_size: 4096 2024-03-11T22:17:41.615 INFO:journalctl@ceph.mon.c.smithi183.stdout: partition_filters: 0 2024-03-11T22:17:41.615 INFO:journalctl@ceph.mon.c.smithi183.stdout: use_delta_encoding: 1 2024-03-11T22:17:41.615 INFO:journalctl@ceph.mon.c.smithi183.stdout: filter_policy: bloomfilter 2024-03-11T22:17:41.615 INFO:journalctl@ceph.mon.c.smithi183.stdout: whole_key_filtering: 1 2024-03-11T22:17:41.615 INFO:journalctl@ceph.mon.c.smithi183.stdout: verify_compression: 0 2024-03-11T22:17:41.615 INFO:journalctl@ceph.mon.c.smithi183.stdout: read_amp_bytes_per_bit: 0 2024-03-11T22:17:41.616 INFO:journalctl@ceph.mon.c.smithi183.stdout: format_version: 5 2024-03-11T22:17:41.616 INFO:journalctl@ceph.mon.c.smithi183.stdout: enable_index_compression: 1 2024-03-11T22:17:41.616 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_align: 0 2024-03-11T22:17:41.616 INFO:journalctl@ceph.mon.c.smithi183.stdout: max_auto_readahead_size: 262144 2024-03-11T22:17:41.616 INFO:journalctl@ceph.mon.c.smithi183.stdout: prepopulate_block_cache: 0 2024-03-11T22:17:41.616 INFO:journalctl@ceph.mon.c.smithi183.stdout: initial_auto_readahead_size: 8192 2024-03-11T22:17:41.616 INFO:journalctl@ceph.mon.c.smithi183.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T22:17:41.616 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T22:17:41.616 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T22:17:41.616 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compression: NoCompression 2024-03-11T22:17:41.616 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T22:17:41.616 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T22:17:41.616 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T22:17:41.616 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.num_levels: 7 2024-03-11T22:17:41.616 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T22:17:41.616 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T22:17:41.617 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T22:17:41.617 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T22:17:41.617 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T22:17:41.617 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T22:17:41.617 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T22:17:41.617 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:17:41.617 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T22:17:41.617 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T22:17:41.617 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:17:41.617 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:17:41.617 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T22:17:41.617 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T22:17:41.617 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T22:17:41.617 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T22:17:41.617 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:17:41.617 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:17:41.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T22:17:41.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T22:17:41.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:17:41.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T22:17:41.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T22:17:41.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T22:17:41.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T22:17:41.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T22:17:41.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T22:17:41.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T22:17:41.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T22:17:41.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T22:17:41.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T22:17:41.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T22:17:41.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T22:17:41.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T22:17:41.619 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T22:17:41.619 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T22:17:41.619 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T22:17:41.619 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T22:17:41.619 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T22:17:41.619 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T22:17:41.619 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T22:17:41.619 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T22:17:41.619 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T22:17:41.619 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T22:17:41.619 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T22:17:41.619 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T22:17:41.619 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T22:17:41.619 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T22:17:41.619 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T22:17:41.620 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T22:17:41.620 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T22:17:41.620 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T22:17:41.620 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T22:17:41.620 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.table_properties_collectors: 2024-03-11T22:17:41.620 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.inplace_update_support: 0 2024-03-11T22:17:41.620 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T22:17:41.620 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T22:17:41.620 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T22:17:41.620 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T22:17:41.620 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.bloom_locality: 0 2024-03-11T22:17:41.620 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.max_successive_merges: 0 2024-03-11T22:17:41.620 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T22:17:41.620 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T22:17:41.620 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T22:17:41.621 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T22:17:41.621 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.ttl: 2592000 2024-03-11T22:17:41.621 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T22:17:41.621 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T22:17:41.621 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T22:17:41.621 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.enable_blob_files: false 2024-03-11T22:17:41.621 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.min_blob_size: 0 2024-03-11T22:17:41.621 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T22:17:41.621 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T22:17:41.621 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T22:17:41.621 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T22:17:41.621 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T22:17:41.621 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T22:17:41.621 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T22:17:41.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T22:17:41.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T22:17:41.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-11T22:17:41.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c1b962dc-5aa7-45d7-90f7-1e8797d6d2b2 2024-03-11T22:17:41.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710195461349704, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-11T22:17:41.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-11T22:17:41.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710195461350112, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710195461, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c1b962dc-5aa7-45d7-90f7-1e8797d6d2b2", "db_session_id": "PI1FTYS64BH3XB6ZLAL8", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-11T22:17:41.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710195461350192, "job": 1, "event": "recovery_finished"} 2024-03-11T22:17:41.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-11T22:17:41.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T22:17:41.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556ac717c000 2024-03-11T22:17:41.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: DB pointer 0x556ac7166000 2024-03-11T22:17:41.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T22:17:41.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T22:17:41.623 INFO:journalctl@ceph.mon.c.smithi183.stdout: ** DB Stats ** 2024-03-11T22:17:41.623 INFO:journalctl@ceph.mon.c.smithi183.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:17:41.623 INFO:journalctl@ceph.mon.c.smithi183.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T22:17:41.623 INFO:journalctl@ceph.mon.c.smithi183.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:17:41.623 INFO:journalctl@ceph.mon.c.smithi183.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:17:41.623 INFO:journalctl@ceph.mon.c.smithi183.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T22:17:41.623 INFO:journalctl@ceph.mon.c.smithi183.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:17:41.623 INFO:journalctl@ceph.mon.c.smithi183.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:17:41.623 INFO:journalctl@ceph.mon.c.smithi183.stdout: 2024-03-11T22:17:41.623 INFO:journalctl@ceph.mon.c.smithi183.stdout: ** Compaction Stats [default] ** 2024-03-11T22:17:41.623 INFO:journalctl@ceph.mon.c.smithi183.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:17:41.623 INFO:journalctl@ceph.mon.c.smithi183.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T22:17:41.623 INFO:journalctl@ceph.mon.c.smithi183.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:17:41.623 INFO:journalctl@ceph.mon.c.smithi183.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:17:41.623 INFO:journalctl@ceph.mon.c.smithi183.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:17:41.624 INFO:journalctl@ceph.mon.c.smithi183.stdout: 2024-03-11T22:17:41.624 INFO:journalctl@ceph.mon.c.smithi183.stdout: ** Compaction Stats [default] ** 2024-03-11T22:17:41.624 INFO:journalctl@ceph.mon.c.smithi183.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:17:41.624 INFO:journalctl@ceph.mon.c.smithi183.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:17:41.624 INFO:journalctl@ceph.mon.c.smithi183.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:17:41.624 INFO:journalctl@ceph.mon.c.smithi183.stdout: 2024-03-11T22:17:41.624 INFO:journalctl@ceph.mon.c.smithi183.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T22:17:41.624 INFO:journalctl@ceph.mon.c.smithi183.stdout: 2024-03-11T22:17:41.624 INFO:journalctl@ceph.mon.c.smithi183.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:17:41.624 INFO:journalctl@ceph.mon.c.smithi183.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T22:17:41.624 INFO:journalctl@ceph.mon.c.smithi183.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T22:17:41.624 INFO:journalctl@ceph.mon.c.smithi183.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T22:17:41.624 INFO:journalctl@ceph.mon.c.smithi183.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T22:17:41.624 INFO:journalctl@ceph.mon.c.smithi183.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T22:17:41.624 INFO:journalctl@ceph.mon.c.smithi183.stdout: Cumulative compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:17:41.624 INFO:journalctl@ceph.mon.c.smithi183.stdout: Interval compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:17:41.624 INFO:journalctl@ceph.mon.c.smithi183.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T22:17:41.625 INFO:journalctl@ceph.mon.c.smithi183.stdout: Block cache BinnedLRUCache@0x556ac708b090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-11T22:17:41.625 INFO:journalctl@ceph.mon.c.smithi183.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-11T22:17:41.625 INFO:journalctl@ceph.mon.c.smithi183.stdout: 2024-03-11T22:17:41.625 INFO:journalctl@ceph.mon.c.smithi183.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T22:17:41.625 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-11T22:17:41.625 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: using public_addr v2:172.21.15.183:0/0 -> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2024-03-11T22:17:41.625 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: starting mon.c rank -1 at public addrs [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] at bind addrs [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:17:41.625 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: mon.c@-1(???) e0 preinit fsid c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:17:41.625 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: mon.c@-1(synchronizing).mds e1 new map 2024-03-11T22:17:41.625 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-11T22:17:41.625 INFO:journalctl@ceph.mon.c.smithi183.stdout: e1 2024-03-11T22:17:41.625 INFO:journalctl@ceph.mon.c.smithi183.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T22:17:41.625 INFO:journalctl@ceph.mon.c.smithi183.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T22:17:41.625 INFO:journalctl@ceph.mon.c.smithi183.stdout: legacy client fscid: -1 2024-03-11T22:17:41.625 INFO:journalctl@ceph.mon.c.smithi183.stdout: 2024-03-11T22:17:41.626 INFO:journalctl@ceph.mon.c.smithi183.stdout: No filesystems configured 2024-03-11T22:17:41.626 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T22:17:41.626 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T22:17:41.626 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T22:17:41.626 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T22:17:41.626 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T22:17:41.626 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T22:17:41.626 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T22:17:41.626 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:17:41.626 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:17:41.626 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:17:41.626 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:41.626 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:41.626 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi183", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:41.626 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:41.627 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:41.627 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:41.627 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:41.627 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:17:41.627 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:41.627 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:17:41.627 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-11T22:17:41.627 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: Deploying cephadm binary to smithi183 2024-03-11T22:17:41.627 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: Updating smithi174:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:17:41.627 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:17:41.627 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: Updating smithi174:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:17:41.627 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:41.627 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:41.627 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:41.627 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:41.627 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:41.628 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:41.628 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: Added host smithi183 2024-03-11T22:17:41.628 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:41.628 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:41.628 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:41.628 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:41.628 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:41.628 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:17:41.628 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:41.628 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/2342907253' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T22:17:41.628 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:41.628 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/2342907253' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T22:17:41.628 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:17:41.628 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:41.628 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:41.628 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:41.628 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:41.629 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:17:41.629 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:41.629 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:17:41.629 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-11T22:17:41.629 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:41.629 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:41.629 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:41.629 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: Updating smithi183:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:17:41.629 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:17:41.629 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: Updating smithi183:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:17:41.629 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:41.629 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:41.629 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:41.629 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:41.629 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:17:41.629 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:41.629 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:17:41.630 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:41.630 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi174:172.21.15.174=b;smithi183:172.21.15.183=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:41.630 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: Saving service mon spec with placement smithi027:172.21.15.27=a;smithi174:172.21.15.174=b;smithi183:172.21.15.183=c;count:3 2024-03-11T22:17:41.630 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: Deploying daemon mon.c on smithi183 2024-03-11T22:17:41.630 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:41.630 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:41.630 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: from='client.? 172.21.15.183:0/1562450803' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:17:41.630 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 ceph-mon[30906]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-11T22:17:41.630 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 bash[30868]: b32fe2121d6f65c953cfe2785f69edf1b058faca64fd7c20c50876fc74fd1ae3 2024-03-11T22:17:41.630 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:41 smithi183 systemd[1]: Started Ceph mon.c for c71f9f14-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:17:41.915 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T22:17:41.915 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:17:42.149 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:17:46.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:17:46.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: mon.a calling monitor election 2024-03-11T22:17:46.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:46.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:46.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:46.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:46.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: mon.c calling monitor election 2024-03-11T22:17:46.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:46.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:46.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:46.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:46.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T22:17:46.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: monmap e2: 2 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:17:46.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: fsmap 2024-03-11T22:17:46.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:17:46.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: mgrmap e13: y(active, since 52s) 2024-03-11T22:17:46.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: overall HEALTH_OK 2024-03-11T22:17:46.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:46.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:46.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:46.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:17:46.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:46 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:46.812 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-11T22:17:46.812 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":2,"fsid":"c71f9f14-dff4-11ee-95c8-87774f69a715","modified":"2024-03-11T22:17:41.374779Z","created":"2024-03-11T22:15:53.875211Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:3300","nonce":0},{"type":"v1","addr":"172.21.15.183:6789","nonce":0}]},"addr":"172.21.15.183:6789/0","public_addr":"172.21.15.183:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T22:17:46.812 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:17:46.812 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: mon.a calling monitor election 2024-03-11T22:17:46.812 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:46.812 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:46.813 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:46.813 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:46.813 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: mon.c calling monitor election 2024-03-11T22:17:46.813 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:46.813 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:46.813 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:46.813 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:46.813 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T22:17:46.813 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: monmap e2: 2 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:17:46.813 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: fsmap 2024-03-11T22:17:46.813 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:17:46.813 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: mgrmap e13: y(active, since 52s) 2024-03-11T22:17:46.813 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: overall HEALTH_OK 2024-03-11T22:17:46.813 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:46.814 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:46.814 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:46.814 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:17:46.814 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:46 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:46.814 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 2 2024-03-11T22:17:47.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:47 smithi183 ceph-mon[30906]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:47.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:47 smithi183 ceph-mon[30906]: Deploying daemon mon.b on smithi174 2024-03-11T22:17:47.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:47 smithi183 ceph-mon[30906]: from='client.? 172.21.15.183:0/1281030571' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:17:47.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:47 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:47.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:47 smithi027 ceph-mon[25797]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:47.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:47 smithi027 ceph-mon[25797]: Deploying daemon mon.b on smithi174 2024-03-11T22:17:47.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:47 smithi027 ceph-mon[25797]: from='client.? 172.21.15.183:0/1281030571' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:17:47.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:47 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:48.354 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T22:17:48.354 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:17:48.598 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:17:49.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:49 smithi027 ceph-mon[25797]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:49.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:49 smithi183 ceph-mon[30906]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:50.159 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 podman[30047]: 2024-03-11 22:17:49.932141223 +0000 UTC m=+0.018001203 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:17:50.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 podman[30047]: 2024-03-11 22:17:50.253414469 +0000 UTC m=+0.339274433 container init c06c2efb9f9ed9e5fb486e2807da25167dcc47ccf79169db1fb109d1b8a1a333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-11T22:17:50.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 podman[30047]: 2024-03-11 22:17:50.259080465 +0000 UTC m=+0.344940430 container start c06c2efb9f9ed9e5fb486e2807da25167dcc47ccf79169db1fb109d1b8a1a333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-b, RELEASE=squid-94a5397, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-11T22:17:50.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T22:17:50.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 7 2024-03-11T22:17:50.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: pidfile_write: ignore empty --pid-file 2024-03-11T22:17:50.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: load: jerasure load: lrc 2024-03-11T22:17:50.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: RocksDB version: 7.9.2 2024-03-11T22:17:50.506 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Git sha 0 2024-03-11T22:17:50.506 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T22:17:50.506 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: DB SUMMARY 2024-03-11T22:17:50.506 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: DB Session ID: 7FA1BLU8NH5ZJY7R04TP 2024-03-11T22:17:50.506 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: CURRENT file: CURRENT 2024-03-11T22:17:50.506 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T22:17:50.507 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-11T22:17:50.507 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-11T22:17:50.507 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-11T22:17:50.507 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.error_if_exists: 0 2024-03-11T22:17:50.507 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.create_if_missing: 0 2024-03-11T22:17:50.507 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.paranoid_checks: 1 2024-03-11T22:17:50.507 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T22:17:50.508 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T22:17:50.508 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T22:17:50.508 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.env: 0x55a490817ae0 2024-03-11T22:17:50.508 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T22:17:50.508 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.info_log: 0x55a4922138c0 2024-03-11T22:17:50.508 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T22:17:50.508 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.statistics: (nil) 2024-03-11T22:17:50.508 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.use_fsync: 0 2024-03-11T22:17:50.508 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_log_file_size: 0 2024-03-11T22:17:50.509 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T22:17:50.509 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T22:17:50.509 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T22:17:50.509 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T22:17:50.509 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.allow_fallocate: 1 2024-03-11T22:17:50.509 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T22:17:50.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T22:17:50.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.use_direct_reads: 0 2024-03-11T22:17:50.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T22:17:50.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T22:17:50.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.db_log_dir: 2024-03-11T22:17:50.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.wal_dir: 2024-03-11T22:17:50.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T22:17:50.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T22:17:50.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T22:17:50.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T22:17:50.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T22:17:50.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T22:17:50.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T22:17:50.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T22:17:50.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.write_buffer_manager: 0x55a4921e7400 2024-03-11T22:17:50.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T22:17:50.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T22:17:50.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T22:17:50.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T22:17:50.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T22:17:50.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T22:17:50.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T22:17:50.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T22:17:50.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.unordered_write: 0 2024-03-11T22:17:50.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T22:17:50.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T22:17:50.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T22:17:50.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T22:17:50.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.row_cache: None 2024-03-11T22:17:50.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.wal_filter: None 2024-03-11T22:17:50.516 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T22:17:50.516 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T22:17:50.516 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.two_write_queues: 0 2024-03-11T22:17:50.516 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T22:17:50.517 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.wal_compression: 0 2024-03-11T22:17:50.517 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.atomic_flush: 0 2024-03-11T22:17:50.517 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T22:17:50.517 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T22:17:50.517 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T22:17:50.517 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.log_readahead_size: 0 2024-03-11T22:17:50.517 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T22:17:50.517 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T22:17:50.518 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T22:17:50.518 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T22:17:50.518 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T22:17:50.518 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T22:17:50.518 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T22:17:50.518 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_background_jobs: 2 2024-03-11T22:17:50.518 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_background_compactions: -1 2024-03-11T22:17:50.519 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_subcompactions: 1 2024-03-11T22:17:50.519 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T22:17:50.519 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T22:17:50.519 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T22:17:50.519 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T22:17:50.519 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T22:17:50.519 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T22:17:50.519 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T22:17:50.520 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T22:17:50.520 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_open_files: -1 2024-03-11T22:17:50.520 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T22:17:50.520 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T22:17:50.520 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T22:17:50.520 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T22:17:50.520 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_background_flushes: -1 2024-03-11T22:17:50.521 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Compression algorithms supported: 2024-03-11T22:17:50.521 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T22:17:50.521 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: kZSTD supported: 0 2024-03-11T22:17:50.521 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: kXpressCompression supported: 0 2024-03-11T22:17:50.521 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T22:17:50.521 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: kZlibCompression supported: 1 2024-03-11T22:17:50.521 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: kSnappyCompression supported: 1 2024-03-11T22:17:50.522 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: kLZ4Compression supported: 1 2024-03-11T22:17:50.522 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: kBZip2Compression supported: 0 2024-03-11T22:17:50.522 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T22:17:50.523 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T22:17:50.523 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-11T22:17:50.523 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T22:17:50.523 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T22:17:50.523 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.merge_operator: 2024-03-11T22:17:50.523 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compaction_filter: None 2024-03-11T22:17:50.523 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T22:17:50.523 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T22:17:50.523 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T22:17:50.523 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T22:17:50.523 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a4922139c0) 2024-03-11T22:17:50.523 INFO:journalctl@ceph.mon.b.smithi174.stdout: cache_index_and_filter_blocks: 1 2024-03-11T22:17:50.524 INFO:journalctl@ceph.mon.b.smithi174.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T22:17:50.524 INFO:journalctl@ceph.mon.b.smithi174.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T22:17:50.524 INFO:journalctl@ceph.mon.b.smithi174.stdout: pin_top_level_index_and_filter: 1 2024-03-11T22:17:50.524 INFO:journalctl@ceph.mon.b.smithi174.stdout: index_type: 0 2024-03-11T22:17:50.524 INFO:journalctl@ceph.mon.b.smithi174.stdout: data_block_index_type: 0 2024-03-11T22:17:50.524 INFO:journalctl@ceph.mon.b.smithi174.stdout: index_shortening: 1 2024-03-11T22:17:50.524 INFO:journalctl@ceph.mon.b.smithi174.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T22:17:50.524 INFO:journalctl@ceph.mon.b.smithi174.stdout: checksum: 4 2024-03-11T22:17:50.524 INFO:journalctl@ceph.mon.b.smithi174.stdout: no_block_cache: 0 2024-03-11T22:17:50.524 INFO:journalctl@ceph.mon.b.smithi174.stdout: block_cache: 0x55a492217090 2024-03-11T22:17:50.524 INFO:journalctl@ceph.mon.b.smithi174.stdout: block_cache_name: BinnedLRUCache 2024-03-11T22:17:50.524 INFO:journalctl@ceph.mon.b.smithi174.stdout: block_cache_options: 2024-03-11T22:17:50.524 INFO:journalctl@ceph.mon.b.smithi174.stdout: capacity : 536870912 2024-03-11T22:17:50.524 INFO:journalctl@ceph.mon.b.smithi174.stdout: num_shard_bits : 4 2024-03-11T22:17:50.524 INFO:journalctl@ceph.mon.b.smithi174.stdout: strict_capacity_limit : 0 2024-03-11T22:17:50.525 INFO:journalctl@ceph.mon.b.smithi174.stdout: high_pri_pool_ratio: 0.000 2024-03-11T22:17:50.525 INFO:journalctl@ceph.mon.b.smithi174.stdout: block_cache_compressed: (nil) 2024-03-11T22:17:50.525 INFO:journalctl@ceph.mon.b.smithi174.stdout: persistent_cache: (nil) 2024-03-11T22:17:50.525 INFO:journalctl@ceph.mon.b.smithi174.stdout: block_size: 4096 2024-03-11T22:17:50.525 INFO:journalctl@ceph.mon.b.smithi174.stdout: block_size_deviation: 10 2024-03-11T22:17:50.525 INFO:journalctl@ceph.mon.b.smithi174.stdout: block_restart_interval: 16 2024-03-11T22:17:50.525 INFO:journalctl@ceph.mon.b.smithi174.stdout: index_block_restart_interval: 1 2024-03-11T22:17:50.525 INFO:journalctl@ceph.mon.b.smithi174.stdout: metadata_block_size: 4096 2024-03-11T22:17:50.525 INFO:journalctl@ceph.mon.b.smithi174.stdout: partition_filters: 0 2024-03-11T22:17:50.525 INFO:journalctl@ceph.mon.b.smithi174.stdout: use_delta_encoding: 1 2024-03-11T22:17:50.525 INFO:journalctl@ceph.mon.b.smithi174.stdout: filter_policy: bloomfilter 2024-03-11T22:17:50.525 INFO:journalctl@ceph.mon.b.smithi174.stdout: whole_key_filtering: 1 2024-03-11T22:17:50.525 INFO:journalctl@ceph.mon.b.smithi174.stdout: verify_compression: 0 2024-03-11T22:17:50.525 INFO:journalctl@ceph.mon.b.smithi174.stdout: read_amp_bytes_per_bit: 0 2024-03-11T22:17:50.526 INFO:journalctl@ceph.mon.b.smithi174.stdout: format_version: 5 2024-03-11T22:17:50.526 INFO:journalctl@ceph.mon.b.smithi174.stdout: enable_index_compression: 1 2024-03-11T22:17:50.526 INFO:journalctl@ceph.mon.b.smithi174.stdout: block_align: 0 2024-03-11T22:17:50.526 INFO:journalctl@ceph.mon.b.smithi174.stdout: max_auto_readahead_size: 262144 2024-03-11T22:17:50.526 INFO:journalctl@ceph.mon.b.smithi174.stdout: prepopulate_block_cache: 0 2024-03-11T22:17:50.526 INFO:journalctl@ceph.mon.b.smithi174.stdout: initial_auto_readahead_size: 8192 2024-03-11T22:17:50.526 INFO:journalctl@ceph.mon.b.smithi174.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T22:17:50.526 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T22:17:50.526 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T22:17:50.526 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compression: NoCompression 2024-03-11T22:17:50.526 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T22:17:50.526 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T22:17:50.527 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T22:17:50.527 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.num_levels: 7 2024-03-11T22:17:50.527 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T22:17:50.527 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T22:17:50.527 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T22:17:50.527 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T22:17:50.527 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T22:17:50.527 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T22:17:50.527 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T22:17:50.527 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:17:50.527 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T22:17:50.527 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T22:17:50.527 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:17:50.527 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:17:50.528 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T22:17:50.528 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T22:17:50.528 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T22:17:50.528 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T22:17:50.528 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:17:50.528 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:17:50.528 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T22:17:50.528 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T22:17:50.528 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:17:50.528 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T22:17:50.528 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T22:17:50.528 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T22:17:50.528 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T22:17:50.528 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T22:17:50.529 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T22:17:50.529 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T22:17:50.529 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T22:17:50.529 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T22:17:50.529 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T22:17:50.529 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T22:17:50.529 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T22:17:50.529 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T22:17:50.529 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T22:17:50.529 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T22:17:50.529 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T22:17:50.529 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T22:17:50.529 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T22:17:50.529 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T22:17:50.530 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T22:17:50.530 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T22:17:50.530 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T22:17:50.530 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T22:17:50.530 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T22:17:50.530 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T22:17:50.530 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T22:17:50.530 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T22:17:50.530 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T22:17:50.530 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T22:17:50.530 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T22:17:50.530 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T22:17:50.530 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T22:17:50.531 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.table_properties_collectors: 2024-03-11T22:17:50.531 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.inplace_update_support: 0 2024-03-11T22:17:50.531 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T22:17:50.531 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T22:17:50.531 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T22:17:50.531 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T22:17:50.531 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.bloom_locality: 0 2024-03-11T22:17:50.531 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.max_successive_merges: 0 2024-03-11T22:17:50.531 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T22:17:50.531 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T22:17:50.531 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T22:17:50.531 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T22:17:50.531 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.ttl: 2592000 2024-03-11T22:17:50.531 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T22:17:50.532 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T22:17:50.532 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T22:17:50.532 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.enable_blob_files: false 2024-03-11T22:17:50.532 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.min_blob_size: 0 2024-03-11T22:17:50.532 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T22:17:50.532 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T22:17:50.532 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T22:17:50.532 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T22:17:50.532 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T22:17:50.532 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T22:17:50.532 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T22:17:50.532 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T22:17:50.532 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T22:17:50.532 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-11T22:17:50.532 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2e3a7c89-3fef-4e88-b54e-0311d2cc9677 2024-03-11T22:17:50.533 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710195470288979, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-11T22:17:50.533 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-11T22:17:50.533 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710195470289392, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710195470, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2e3a7c89-3fef-4e88-b54e-0311d2cc9677", "db_session_id": "7FA1BLU8NH5ZJY7R04TP", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-11T22:17:50.533 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710195470289471, "job": 1, "event": "recovery_finished"} 2024-03-11T22:17:50.533 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-11T22:17:50.533 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T22:17:50.533 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a492308000 2024-03-11T22:17:50.533 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: DB pointer 0x55a4922f2000 2024-03-11T22:17:50.533 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T22:17:50.533 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T22:17:50.533 INFO:journalctl@ceph.mon.b.smithi174.stdout: ** DB Stats ** 2024-03-11T22:17:50.533 INFO:journalctl@ceph.mon.b.smithi174.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:17:50.533 INFO:journalctl@ceph.mon.b.smithi174.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T22:17:50.533 INFO:journalctl@ceph.mon.b.smithi174.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:17:50.533 INFO:journalctl@ceph.mon.b.smithi174.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:17:50.533 INFO:journalctl@ceph.mon.b.smithi174.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T22:17:50.534 INFO:journalctl@ceph.mon.b.smithi174.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:17:50.534 INFO:journalctl@ceph.mon.b.smithi174.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:17:50.534 INFO:journalctl@ceph.mon.b.smithi174.stdout: 2024-03-11T22:17:50.534 INFO:journalctl@ceph.mon.b.smithi174.stdout: ** Compaction Stats [default] ** 2024-03-11T22:17:50.534 INFO:journalctl@ceph.mon.b.smithi174.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:17:50.534 INFO:journalctl@ceph.mon.b.smithi174.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T22:17:50.534 INFO:journalctl@ceph.mon.b.smithi174.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:17:50.534 INFO:journalctl@ceph.mon.b.smithi174.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:17:50.534 INFO:journalctl@ceph.mon.b.smithi174.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:17:50.534 INFO:journalctl@ceph.mon.b.smithi174.stdout: 2024-03-11T22:17:50.534 INFO:journalctl@ceph.mon.b.smithi174.stdout: ** Compaction Stats [default] ** 2024-03-11T22:17:50.534 INFO:journalctl@ceph.mon.b.smithi174.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:17:50.534 INFO:journalctl@ceph.mon.b.smithi174.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:17:50.534 INFO:journalctl@ceph.mon.b.smithi174.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:17:50.535 INFO:journalctl@ceph.mon.b.smithi174.stdout: 2024-03-11T22:17:50.535 INFO:journalctl@ceph.mon.b.smithi174.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T22:17:50.535 INFO:journalctl@ceph.mon.b.smithi174.stdout: 2024-03-11T22:17:50.535 INFO:journalctl@ceph.mon.b.smithi174.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:17:50.535 INFO:journalctl@ceph.mon.b.smithi174.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T22:17:50.535 INFO:journalctl@ceph.mon.b.smithi174.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T22:17:50.535 INFO:journalctl@ceph.mon.b.smithi174.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T22:17:50.535 INFO:journalctl@ceph.mon.b.smithi174.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T22:17:50.535 INFO:journalctl@ceph.mon.b.smithi174.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T22:17:50.535 INFO:journalctl@ceph.mon.b.smithi174.stdout: Cumulative compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:17:50.535 INFO:journalctl@ceph.mon.b.smithi174.stdout: Interval compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:17:50.535 INFO:journalctl@ceph.mon.b.smithi174.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T22:17:50.535 INFO:journalctl@ceph.mon.b.smithi174.stdout: Block cache BinnedLRUCache@0x55a492217090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-11T22:17:50.535 INFO:journalctl@ceph.mon.b.smithi174.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-11T22:17:50.535 INFO:journalctl@ceph.mon.b.smithi174.stdout: 2024-03-11T22:17:50.535 INFO:journalctl@ceph.mon.b.smithi174.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T22:17:50.536 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-11T22:17:50.536 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: using public_addr v2:172.21.15.174:0/0 -> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-03-11T22:17:50.536 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: starting mon.b rank -1 at public addrs [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] at bind addrs [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:17:50.536 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: mon.b@-1(???) e0 preinit fsid c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:17:50.536 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: mon.b@-1(synchronizing).mds e1 new map 2024-03-11T22:17:50.536 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-11T22:17:50.536 INFO:journalctl@ceph.mon.b.smithi174.stdout: e1 2024-03-11T22:17:50.536 INFO:journalctl@ceph.mon.b.smithi174.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T22:17:50.536 INFO:journalctl@ceph.mon.b.smithi174.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T22:17:50.536 INFO:journalctl@ceph.mon.b.smithi174.stdout: legacy client fscid: -1 2024-03-11T22:17:50.536 INFO:journalctl@ceph.mon.b.smithi174.stdout: 2024-03-11T22:17:50.536 INFO:journalctl@ceph.mon.b.smithi174.stdout: No filesystems configured 2024-03-11T22:17:50.536 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T22:17:50.536 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T22:17:50.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T22:17:50.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T22:17:50.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T22:17:50.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T22:17:50.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T22:17:50.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:17:50.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:17:50.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:17:50.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:50.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:50.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi183", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:50.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:17:50.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:50.538 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:17:50.538 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-11T22:17:50.538 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: Deploying cephadm binary to smithi183 2024-03-11T22:17:50.538 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: Updating smithi174:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:17:50.538 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:17:50.538 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: Updating smithi174:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:17:50.538 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:50.538 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:50.538 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.538 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.538 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.538 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.538 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: Added host smithi183 2024-03-11T22:17:50.538 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:50.539 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.539 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.539 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:50.539 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.539 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:17:50.539 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:50.539 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/2342907253' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T22:17:50.539 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:50.539 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/2342907253' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T22:17:50.539 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:17:50.539 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.539 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.539 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.539 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.539 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:17:50.539 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:50.540 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:17:50.540 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-11T22:17:50.540 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.540 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.540 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.540 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: Updating smithi183:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:17:50.540 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:17:50.540 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: Updating smithi183:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:17:50.540 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:50.540 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.540 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:50.540 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:50.540 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:17:50.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:17:50.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:50.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi174:172.21.15.174=b;smithi183:172.21.15.183=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:50.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: Saving service mon spec with placement smithi027:172.21.15.27=a;smithi174:172.21.15.174=b;smithi183:172.21.15.183=c;count:3 2024-03-11T22:17:50.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: Deploying daemon mon.c on smithi183 2024-03-11T22:17:50.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:50.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:50.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='client.? 172.21.15.183:0/1562450803' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:17:50.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:17:50.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: mon.a calling monitor election 2024-03-11T22:17:50.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:50.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:50.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:50.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:50.542 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: mon.c calling monitor election 2024-03-11T22:17:50.542 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:50.542 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:50.542 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:50.542 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:50.542 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T22:17:50.542 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: monmap e2: 2 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:17:50.542 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: fsmap 2024-03-11T22:17:50.542 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:17:50.542 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: mgrmap e13: y(active, since 52s) 2024-03-11T22:17:50.542 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: overall HEALTH_OK 2024-03-11T22:17:50.542 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.542 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.542 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:50.543 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:17:50.543 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:50.543 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:50.543 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: Deploying daemon mon.b on smithi174 2024-03-11T22:17:50.543 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='client.? 172.21.15.183:0/1281030571' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:17:50.543 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:50.543 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:50.543 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 ceph-mon[30085]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-11T22:17:50.543 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 bash[30047]: c06c2efb9f9ed9e5fb486e2807da25167dcc47ccf79169db1fb109d1b8a1a333 2024-03-11T22:17:50.543 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:50 smithi174 systemd[1]: Started Ceph mon.b for c71f9f14-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:17:55.346 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-11T22:17:55.347 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":3,"fsid":"c71f9f14-dff4-11ee-95c8-87774f69a715","modified":"2024-03-11T22:17:50.312367Z","created":"2024-03-11T22:15:53.875211Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:3300","nonce":0},{"type":"v1","addr":"172.21.15.183:6789","nonce":0}]},"addr":"172.21.15.183:6789/0","public_addr":"172.21.15.183:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T22:17:55.349 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 3 2024-03-11T22:17:55.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:17:55.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: mon.c calling monitor election 2024-03-11T22:17:55.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: mon.a calling monitor election 2024-03-11T22:17:55.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:55.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:55.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:55.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: from='client.? 172.21.15.183:0/870906267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:17:55.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:55.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:55.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:55.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:55.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:55.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:55.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:55.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T22:17:55.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: monmap e3: 3 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],b=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:17:55.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: fsmap 2024-03-11T22:17:55.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:17:55.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: mgrmap e13: y(active, since 60s) 2024-03-11T22:17:55.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T22:17:55.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T22:17:55.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T22:17:55.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: mon.b (rank 2) addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] is down (out of quorum) 2024-03-11T22:17:55.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:55.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:55.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:55.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:55 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:55.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:17:55.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: mon.c calling monitor election 2024-03-11T22:17:55.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: mon.a calling monitor election 2024-03-11T22:17:55.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:55.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:55.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:55.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: from='client.? 172.21.15.183:0/870906267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:17:55.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:55.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:55.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:55.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:55.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:55.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:55.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:55.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T22:17:55.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: monmap e3: 3 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],b=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:17:55.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: fsmap 2024-03-11T22:17:55.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:17:55.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: mgrmap e13: y(active, since 60s) 2024-03-11T22:17:55.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T22:17:55.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T22:17:55.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T22:17:55.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: mon.b (rank 2) addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] is down (out of quorum) 2024-03-11T22:17:55.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:55.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:55.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:55.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:55 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:55.941 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-11T22:17:55.942 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-11T22:17:56.201 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:17:56.598 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:56 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:56.598 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:56 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:56.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:56 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:56.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:56 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:57.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:17:57.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: mon.c calling monitor election 2024-03-11T22:17:57.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: mon.a calling monitor election 2024-03-11T22:17:57.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:57.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:17:57.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:57.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='client.? 172.21.15.183:0/870906267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:17:57.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:57.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:57.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:57.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:57.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:57.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:57.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:57.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T22:17:57.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: monmap e3: 3 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],b=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:17:57.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: fsmap 2024-03-11T22:17:57.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:17:57.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: mgrmap e13: y(active, since 60s) 2024-03-11T22:17:57.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T22:17:57.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T22:17:57.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T22:17:57.756 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: mon.b (rank 2) addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] is down (out of quorum) 2024-03-11T22:17:57.756 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:57.756 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:57.756 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:57.756 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:57.756 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:57.756 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:58.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: mon.b calling monitor election 2024-03-11T22:17:58.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: mon.b calling monitor election 2024-03-11T22:17:58.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: mon.a calling monitor election 2024-03-11T22:17:58.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: mon.c calling monitor election 2024-03-11T22:17:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T22:17:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: monmap e3: 3 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],b=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:17:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: fsmap 2024-03-11T22:17:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:17:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: mgrmap e13: y(active, since 62s) 2024-03-11T22:17:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T22:17:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: Cluster is now healthy 2024-03-11T22:17:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: overall HEALTH_OK 2024-03-11T22:17:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:17:58.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:17:58.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:58.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: mon.b calling monitor election 2024-03-11T22:17:58.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: mon.b calling monitor election 2024-03-11T22:17:58.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: mon.a calling monitor election 2024-03-11T22:17:58.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: mon.c calling monitor election 2024-03-11T22:17:58.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T22:17:58.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: monmap e3: 3 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],b=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:17:58.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: fsmap 2024-03-11T22:17:58.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:17:58.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: mgrmap e13: y(active, since 62s) 2024-03-11T22:17:58.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T22:17:58.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: Cluster is now healthy 2024-03-11T22:17:58.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: overall HEALTH_OK 2024-03-11T22:17:58.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:17:58.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:17:58.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:58.135 INFO:teuthology.orchestra.run.smithi027.stdout:# minimal ceph.conf for c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:17:58.135 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-03-11T22:17:58.135 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:17:58.135 INFO:teuthology.orchestra.run.smithi027.stdout: mon_host = [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2024-03-11T22:17:58.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: mon.b calling monitor election 2024-03-11T22:17:58.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: mon.b calling monitor election 2024-03-11T22:17:58.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: mon.a calling monitor election 2024-03-11T22:17:58.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: mon.c calling monitor election 2024-03-11T22:17:58.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T22:17:58.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: monmap e3: 3 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],b=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:17:58.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: fsmap 2024-03-11T22:17:58.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:17:58.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: mgrmap e13: y(active, since 62s) 2024-03-11T22:17:58.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T22:17:58.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: Cluster is now healthy 2024-03-11T22:17:58.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: overall HEALTH_OK 2024-03-11T22:17:58.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:17:58.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:17:58.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:17:58.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:59.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:58 smithi183 ceph-mon[30906]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T22:17:59.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:58 smithi183 ceph-mon[30906]: Reconfiguring daemon mon.a on smithi027 2024-03-11T22:17:59.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:58 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/2252391281' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:59.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:58 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:59.163 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:58 smithi027 ceph-mon[25797]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T22:17:59.163 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:58 smithi027 ceph-mon[25797]: Reconfiguring daemon mon.a on smithi027 2024-03-11T22:17:59.163 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:58 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/2252391281' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:59.163 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:58 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:59.196 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-11T22:17:59.196 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-11T22:17:59.196 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T22:17:59.233 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-11T22:17:59.233 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:17:59.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:58 smithi174 ceph-mon[30085]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T22:17:59.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:58 smithi174 ceph-mon[30085]: Reconfiguring daemon mon.a on smithi027 2024-03-11T22:17:59.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:58 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/2252391281' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:59.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:58 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:17:59.306 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-11T22:17:59.307 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T22:17:59.341 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-11T22:17:59.341 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:17:59.411 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-11T22:17:59.411 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T22:17:59.445 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-11T22:17:59.445 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:17:59.516 INFO:tasks.cephadm:Adding mgr.y on smithi027 2024-03-11T22:17:59.516 INFO:tasks.cephadm:Adding mgr.x on smithi174 2024-03-11T22:17:59.516 INFO:tasks.cephadm:Adding mgr.z on smithi183 2024-03-11T22:17:59.517 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph orch apply mgr '3;smithi027=y;smithi174=x;smithi183=z' 2024-03-11T22:17:59.789 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:18:00.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:59 smithi027 ceph-mon[25797]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:00.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:59 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:00.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:59 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:00.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:59 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:00.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:59 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:18:00.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:59 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:18:00.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:17:59 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:00.106 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:59 smithi183 ceph-mon[30906]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:00.106 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:59 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:00.107 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:59 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:00.107 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:59 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:00.107 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:59 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:18:00.107 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:59 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:18:00.107 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:17:59 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:00.253 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:59 smithi174 ceph-mon[30085]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:00.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:59 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:00.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:59 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:00.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:59 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:00.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:59 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:18:00.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:59 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:18:00.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:17:59 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:00.855 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:00 smithi183 ceph-mon[30906]: Reconfiguring mon.b (monmap changed)... 2024-03-11T22:18:00.856 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:00 smithi183 ceph-mon[30906]: Reconfiguring daemon mon.b on smithi174 2024-03-11T22:18:01.245 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:00 smithi174 ceph-mon[30085]: Reconfiguring mon.b (monmap changed)... 2024-03-11T22:18:01.245 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:00 smithi174 ceph-mon[30085]: Reconfiguring daemon mon.b on smithi174 2024-03-11T22:18:01.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:00 smithi027 ceph-mon[25797]: Reconfiguring mon.b (monmap changed)... 2024-03-11T22:18:01.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:00 smithi027 ceph-mon[25797]: Reconfiguring daemon mon.b on smithi174 2024-03-11T22:18:01.553 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled mgr update... 2024-03-11T22:18:02.138 DEBUG:teuthology.orchestra.run.smithi174:mgr.x> sudo journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mgr.x.service 2024-03-11T22:18:02.140 DEBUG:teuthology.orchestra.run.smithi183:mgr.z> sudo journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mgr.z.service 2024-03-11T22:18:02.143 INFO:tasks.cephadm:Deploying OSDs... 2024-03-11T22:18:02.143 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-11T22:18:02.143 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:18:02.162 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T22:18:02.163 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/nvme1n1 2024-03-11T22:18:02.222 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/nvme1n1 2024-03-11T22:18:02.223 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:18:02.223 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 763 Links: 1 Device type: 103,2 2024-03-11T22:18:02.223 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:18:02.223 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:18:02.223 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-03-11 22:17:07.553266995 +0000 2024-03-11T22:18:02.223 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-11 22:14:46.101747769 +0000 2024-03-11T22:18:02.223 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-11 22:14:46.101747769 +0000 2024-03-11T22:18:02.223 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-03-11 22:14:46.091748016 +0000 2024-03-11T22:18:02.223 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-11T22:18:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-03-11T22:18:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-03-11T22:18:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000302767 s, 1.7 MB/s 2024-03-11T22:18:02.299 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-11T22:18:02.358 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/nvme2n1 2024-03-11T22:18:02.415 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/nvme2n1 2024-03-11T22:18:02.415 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:18:02.415 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 776 Links: 1 Device type: 103,4 2024-03-11T22:18:02.415 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:18:02.415 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:18:02.416 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-03-11 22:17:07.564266727 +0000 2024-03-11T22:18:02.416 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-11 22:14:46.237744418 +0000 2024-03-11T22:18:02.416 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-11 22:14:46.237744418 +0000 2024-03-11T22:18:02.416 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-03-11 22:14:46.229744615 +0000 2024-03-11T22:18:02.416 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-11T22:18:02.472 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:02 smithi183 ceph-mon[30906]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:02.473 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:02 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:02.473 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:02 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:02.473 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:02 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:18:02.473 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:02 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:18:02.473 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:02 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:02.473 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:02 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:02.487 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-03-11T22:18:02.488 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-03-11T22:18:02.488 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000248891 s, 2.1 MB/s 2024-03-11T22:18:02.489 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-11T22:18:02.503 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:02 smithi174 ceph-mon[30085]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:02.503 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:02 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:02.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:02 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:02.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:02 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:18:02.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:02 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:18:02.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:02 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:02.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:02 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:02.547 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/nvme3n1 2024-03-11T22:18:02.603 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/nvme3n1 2024-03-11T22:18:02.604 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:18:02.604 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 786 Links: 1 Device type: 103,6 2024-03-11T22:18:02.604 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:18:02.604 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:18:02.604 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-03-11 22:17:07.575266460 +0000 2024-03-11T22:18:02.604 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-11 22:14:46.379740918 +0000 2024-03-11T22:18:02.604 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-11 22:14:46.379740918 +0000 2024-03-11T22:18:02.604 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-03-11 22:14:46.366741239 +0000 2024-03-11T22:18:02.604 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-11T22:18:02.673 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:02 smithi027 ceph-mon[25797]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:02.673 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:02 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:02.673 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:02 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:02.673 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:02 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:18:02.673 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:02 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:18:02.673 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:02 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:02.673 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:02 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:02.676 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-03-11T22:18:02.676 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-03-11T22:18:02.676 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000247902 s, 2.1 MB/s 2024-03-11T22:18:02.677 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-11T22:18:02.735 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/nvme4n1 2024-03-11T22:18:02.790 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/nvme4n1 2024-03-11T22:18:02.790 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:18:02.790 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-11T22:18:02.791 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:18:02.791 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:18:02.791 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-03-11 22:17:07.586266193 +0000 2024-03-11T22:18:02.791 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-11 22:14:46.510737690 +0000 2024-03-11T22:18:02.791 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-11 22:14:46.510737690 +0000 2024-03-11T22:18:02.791 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-03-11 22:14:46.498737986 +0000 2024-03-11T22:18:02.791 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-11T22:18:02.860 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-03-11T22:18:02.860 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-03-11T22:18:02.860 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.00020069 s, 2.6 MB/s 2024-03-11T22:18:02.861 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-11T22:18:02.917 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-11T22:18:02.917 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:18:02.936 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T22:18:02.936 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/nvme1n1 2024-03-11T22:18:02.995 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/nvme1n1 2024-03-11T22:18:02.995 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:18:02.995 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 764 Links: 1 Device type: 103,2 2024-03-11T22:18:02.995 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:18:02.995 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:18:02.996 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-11 22:17:22.285275695 +0000 2024-03-11T22:18:02.996 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-11 22:14:47.702332649 +0000 2024-03-11T22:18:02.996 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-11 22:14:47.702332649 +0000 2024-03-11T22:18:02.996 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-11 22:14:47.694332756 +0000 2024-03-11T22:18:02.996 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-11T22:18:03.065 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-11T22:18:03.065 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-11T22:18:03.065 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000251629 s, 2.0 MB/s 2024-03-11T22:18:03.066 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-11T22:18:03.125 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/nvme2n1 2024-03-11T22:18:03.181 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/nvme2n1 2024-03-11T22:18:03.182 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:18:03.182 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-03-11T22:18:03.182 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:18:03.182 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:18:03.182 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-11 22:17:22.295275563 +0000 2024-03-11T22:18:03.182 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-11 22:14:47.868330416 +0000 2024-03-11T22:18:03.182 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-11 22:14:47.868330416 +0000 2024-03-11T22:18:03.182 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-11 22:14:47.856330577 +0000 2024-03-11T22:18:03.182 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-11T22:18:03.249 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-11T22:18:03.249 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-11T22:18:03.249 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000245006 s, 2.1 MB/s 2024-03-11T22:18:03.251 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-11T22:18:03.307 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/nvme3n1 2024-03-11T22:18:03.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:03 smithi183 ceph-mon[30906]: Reconfiguring mon.c (monmap changed)... 2024-03-11T22:18:03.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:03 smithi183 ceph-mon[30906]: Reconfiguring daemon mon.c on smithi183 2024-03-11T22:18:03.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:03 smithi183 ceph-mon[30906]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=y;smithi174=x;smithi183=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:18:03.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:03 smithi183 ceph-mon[30906]: Saving service mgr spec with placement smithi027=y;smithi174=x;smithi183=z;count:3 2024-03-11T22:18:03.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:03 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:03.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:03 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:03.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:03 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:03.362 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/nvme3n1 2024-03-11T22:18:03.363 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:18:03.363 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-11T22:18:03.363 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:18:03.363 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:18:03.363 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-11 22:17:22.306275418 +0000 2024-03-11T22:18:03.363 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-11 22:14:47.977328949 +0000 2024-03-11T22:18:03.363 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-11 22:14:47.977328949 +0000 2024-03-11T22:18:03.363 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-11 22:14:47.966329097 +0000 2024-03-11T22:18:03.363 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-11T22:18:03.426 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:03 smithi174 ceph-mon[30085]: Reconfiguring mon.c (monmap changed)... 2024-03-11T22:18:03.426 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:03 smithi174 ceph-mon[30085]: Reconfiguring daemon mon.c on smithi183 2024-03-11T22:18:03.426 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:03 smithi174 ceph-mon[30085]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=y;smithi174=x;smithi183=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:18:03.426 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:03 smithi174 ceph-mon[30085]: Saving service mgr spec with placement smithi027=y;smithi174=x;smithi183=z;count:3 2024-03-11T22:18:03.426 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:03 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:03.426 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:03 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:03.427 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:03 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:03.428 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-11T22:18:03.428 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-11T22:18:03.428 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000212858 s, 2.4 MB/s 2024-03-11T22:18:03.430 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-11T22:18:03.486 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/nvme4n1 2024-03-11T22:18:03.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:03 smithi027 ceph-mon[25797]: Reconfiguring mon.c (monmap changed)... 2024-03-11T22:18:03.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:03 smithi027 ceph-mon[25797]: Reconfiguring daemon mon.c on smithi183 2024-03-11T22:18:03.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:03 smithi027 ceph-mon[25797]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=y;smithi174=x;smithi183=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:18:03.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:03 smithi027 ceph-mon[25797]: Saving service mgr spec with placement smithi027=y;smithi174=x;smithi183=z;count:3 2024-03-11T22:18:03.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:03 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:03.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:03 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:03.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:03 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:03.541 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/nvme4n1 2024-03-11T22:18:03.541 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:18:03.541 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 798 Links: 1 Device type: 103,8 2024-03-11T22:18:03.541 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:18:03.541 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:18:03.541 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-11 22:17:22.317275273 +0000 2024-03-11T22:18:03.542 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-11 22:14:48.119327039 +0000 2024-03-11T22:18:03.542 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-11 22:14:48.119327039 +0000 2024-03-11T22:18:03.542 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-11 22:14:48.114327106 +0000 2024-03-11T22:18:03.542 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-11T22:18:03.606 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-11T22:18:03.606 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-11T22:18:03.607 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000239314 s, 2.1 MB/s 2024-03-11T22:18:03.608 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-11T22:18:03.664 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-11T22:18:03.665 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:18:03.681 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T22:18:03.682 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/nvme1n1 2024-03-11T22:18:03.738 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/nvme1n1 2024-03-11T22:18:03.738 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:18:03.738 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-11T22:18:03.738 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:18:03.738 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:18:03.738 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-11 22:17:32.114155424 +0000 2024-03-11T22:18:03.738 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-11 22:14:49.332459693 +0000 2024-03-11T22:18:03.738 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-11 22:14:49.332459693 +0000 2024-03-11T22:18:03.738 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-11 22:14:49.321459988 +0000 2024-03-11T22:18:03.739 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-11T22:18:03.805 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-11T22:18:03.805 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-11T22:18:03.805 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000233118 s, 2.2 MB/s 2024-03-11T22:18:03.806 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-11T22:18:03.863 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/nvme2n1 2024-03-11T22:18:03.917 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/nvme2n1 2024-03-11T22:18:03.917 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:18:03.917 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-11T22:18:03.917 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:18:03.918 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:18:03.918 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-11 22:17:32.125155140 +0000 2024-03-11T22:18:03.918 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-11 22:14:49.487455532 +0000 2024-03-11T22:18:03.918 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-11 22:14:49.487455532 +0000 2024-03-11T22:18:03.918 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-11 22:14:49.478455774 +0000 2024-03-11T22:18:03.918 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-11T22:18:03.983 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-11T22:18:03.983 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-11T22:18:03.984 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000234169 s, 2.2 MB/s 2024-03-11T22:18:03.985 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-11T22:18:04.040 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/nvme3n1 2024-03-11T22:18:04.095 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/nvme3n1 2024-03-11T22:18:04.095 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:18:04.096 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-11T22:18:04.096 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:18:04.096 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:18:04.096 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-11 22:17:32.135154882 +0000 2024-03-11T22:18:04.096 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-11 22:14:49.638451479 +0000 2024-03-11T22:18:04.096 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-11 22:14:49.638451479 +0000 2024-03-11T22:18:04.096 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-11 22:14:49.627451774 +0000 2024-03-11T22:18:04.096 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-11T22:18:04.161 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-11T22:18:04.161 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-11T22:18:04.161 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000219142 s, 2.3 MB/s 2024-03-11T22:18:04.163 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-11T22:18:04.220 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/nvme4n1 2024-03-11T22:18:04.275 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/nvme4n1 2024-03-11T22:18:04.275 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:18:04.275 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-11T22:18:04.275 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:18:04.275 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:18:04.275 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-11 22:17:32.146154599 +0000 2024-03-11T22:18:04.275 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-11 22:14:49.793447318 +0000 2024-03-11T22:18:04.276 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-11 22:14:49.793447318 +0000 2024-03-11T22:18:04.276 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-11 22:14:49.786447506 +0000 2024-03-11T22:18:04.276 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-11T22:18:04.343 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-11T22:18:04.343 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-11T22:18:04.343 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000241563 s, 2.1 MB/s 2024-03-11T22:18:04.344 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-11T22:18:04.357 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:04 smithi183 ceph-mon[30906]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:04.357 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:04 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:04.358 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:04 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:04.358 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:04 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:04.358 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:04 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:18:04.358 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:04 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:18:04.358 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:04 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:18:04.358 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:04 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:04.358 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:04 smithi183 ceph-mon[30906]: Deploying daemon mgr.z on smithi183 2024-03-11T22:18:04.372 INFO:tasks.cephadm:Deploying osd.0 on smithi027 with /dev/nvme4n1... 2024-03-11T22:18:04.373 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-11T22:18:04.622 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:04.742 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:04 smithi027 ceph-mon[25797]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:04.742 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:04 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:04.742 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:04 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:04.742 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:04 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:04.742 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:04 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:18:04.742 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:04 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:18:04.742 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:04 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:18:04.742 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:04 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:04.743 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:04 smithi027 ceph-mon[25797]: Deploying daemon mgr.z on smithi183 2024-03-11T22:18:04.753 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:04 smithi174 ceph-mon[30085]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:04.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:04 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:04.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:04 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:04.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:04 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:04.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:04 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:18:04.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:04 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:18:04.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:04 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:18:04.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:04 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:04.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:04 smithi174 ceph-mon[30085]: Deploying daemon mgr.z on smithi183 2024-03-11T22:18:05.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:05 smithi183 ceph-mon[30906]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:05.604 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:05 smithi027 ceph-mon[25797]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:05.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:05 smithi174 ceph-mon[30085]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:06.058 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:05 smithi183 podman[32965]: 2024-03-11 22:18:05.934322541 +0000 UTC m=+0.117441790 container create 6342b9a9365073a1b55bfc74b1185398f7b6b8b1dbc3b9c9c27d55782b274f93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-11T22:18:06.419 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/nvme4n1 2024-03-11T22:18:06.419 INFO:teuthology.orchestra.run.smithi027.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T22:18:06.419 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-11T22:18:06.419 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-03-11T22:18:06.420 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-03-11T22:18:06.420 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0137888 s, 760 MB/s 2024-03-11T22:18:06.420 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-03-11T22:18:06.603 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:06 smithi183 podman[32965]: 2024-03-11 22:18:06.161662778 +0000 UTC m=+0.344782037 container init 6342b9a9365073a1b55bfc74b1185398f7b6b8b1dbc3b9c9c27d55782b274f93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-11T22:18:06.603 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:06 smithi183 podman[32965]: 2024-03-11 22:18:06.166965542 +0000 UTC m=+0.350084791 container start 6342b9a9365073a1b55bfc74b1185398f7b6b8b1dbc3b9c9c27d55782b274f93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True) 2024-03-11T22:18:06.603 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:06 smithi183 ceph-mgr[33003]: -- 172.21.15.183:0/2301679637 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ac2856a1a0 con 0x55ac28540800 2024-03-11T22:18:06.603 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:06 smithi183 bash[32965]: 6342b9a9365073a1b55bfc74b1185398f7b6b8b1dbc3b9c9c27d55782b274f93 2024-03-11T22:18:06.603 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:06 smithi183 systemd[1]: Started Ceph mgr.z for c71f9f14-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:18:07.031 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi027:/dev/nvme4n1 2024-03-11T22:18:07.103 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:06 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:06.714+0000 7f00589c0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:18:07.103 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:06 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:06.806+0000 7f00589c0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:18:07.278 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:07.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:07 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:07.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:07 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:07.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:07 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:07.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:07 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:18:07.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:07 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:18:07.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:07 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:18:07.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:07 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:07.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:07 smithi174 ceph-mon[30085]: Deploying daemon mgr.x on smithi174 2024-03-11T22:18:07.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:07 smithi174 ceph-mon[30085]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:07.603 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:07 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:07.441+0000 7f00589c0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:18:07.603 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:07 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:07.532+0000 7f00589c0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:18:07.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:07 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:07.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:07 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:07.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:07 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:07.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:07 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:18:07.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:07 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:18:07.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:07 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:18:07.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:07 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:07.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:07 smithi183 ceph-mon[30906]: Deploying daemon mgr.x on smithi174 2024-03-11T22:18:07.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:07 smithi183 ceph-mon[30906]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:07.638 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:07 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:07.638 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:07 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:07.638 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:07 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:07.638 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:07 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:18:07.638 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:07 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:18:07.639 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:07 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:18:07.639 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:07 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:07.639 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:07 smithi027 ceph-mon[25797]: Deploying daemon mgr.x on smithi174 2024-03-11T22:18:07.639 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:07 smithi027 ceph-mon[25797]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:08.103 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:07 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:07.673+0000 7f00589c0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:18:08.103 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:07 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:07.767+0000 7f00589c0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:18:08.103 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:07 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:07.849+0000 7f00589c0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:18:08.397 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:08 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:08.110+0000 7f00589c0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:18:08.397 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:08 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:08.218+0000 7f00589c0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:18:08.397 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:08 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:08.302+0000 7f00589c0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:18:08.397 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:08 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:08.396+0000 7f00589c0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:18:08.643 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:08 smithi174 podman[31521]: 2024-03-11 22:18:08.472457439 +0000 UTC m=+0.019081933 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:18:08.853 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:08 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:08.572+0000 7f00589c0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:18:09.004 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:08 smithi174 podman[31521]: 2024-03-11 22:18:08.737605218 +0000 UTC m=+0.284229706 container init aea4c8612d5dd3874a9bb1d14f5d2d6bcc37346dd8e3af3b7bc669c5e517a979 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-11T22:18:09.004 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:08 smithi174 podman[31521]: 2024-03-11 22:18:08.742685152 +0000 UTC m=+0.289309641 container start aea4c8612d5dd3874a9bb1d14f5d2d6bcc37346dd8e3af3b7bc669c5e517a979 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True) 2024-03-11T22:18:09.004 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:08 smithi174 ceph-mgr[31559]: -- 172.21.15.174:0/3588570643 <== mon.2 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5637611ba1a0 con 0x563761190c00 2024-03-11T22:18:09.004 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:08 smithi174 bash[31521]: aea4c8612d5dd3874a9bb1d14f5d2d6bcc37346dd8e3af3b7bc669c5e517a979 2024-03-11T22:18:09.004 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:08 smithi174 systemd[1]: Started Ceph mgr.x for c71f9f14-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:18:09.754 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:09 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:09.311+0000 7f36c4c94200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:18:09.754 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:09 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:09.412+0000 7f36c4c94200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:18:09.853 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:09 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:09.551+0000 7f00589c0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:18:09.853 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:09 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:09.700+0000 7f00589c0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:18:10.172 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:10 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:10.049+0000 7f00589c0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:18:10.172 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:10 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:10.142+0000 7f00589c0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:18:10.172 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:09 smithi183 ceph-mon[30906]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:10.172 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:09 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:10.172 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:09 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:10.172 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:09 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:10.172 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:09 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:10.173 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:09 smithi183 ceph-mon[30906]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:18:10.173 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:09 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:18:10.173 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:09 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:18:10.173 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:09 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:10.173 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:09 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:10.173 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:09 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:10.173 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:10 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:10.147+0000 7f36c4c94200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:18:10.173 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:09 smithi174 ceph-mon[30085]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:10.173 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:09 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:10.174 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:09 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:10.174 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:09 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:10.174 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:09 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:10.174 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:09 smithi174 ceph-mon[30085]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:18:10.174 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:09 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:18:10.174 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:09 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:18:10.174 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:09 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:10.174 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:09 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:10.174 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:09 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:10.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:09 smithi027 ceph-mon[25797]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:10.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:09 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:10.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:09 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:10.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:09 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:10.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:09 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:10.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:09 smithi027 ceph-mon[25797]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:18:10.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:09 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:18:10.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:09 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:18:10.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:09 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:10.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:09 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:10.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:09 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:10.492 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:10 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:10.242+0000 7f36c4c94200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:18:10.492 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:10 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:10.387+0000 7f36c4c94200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:18:10.492 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:10 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:10.491+0000 7f36c4c94200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:18:10.528 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:10 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:10.336+0000 7f00589c0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:18:10.529 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:10 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:10.527+0000 7f00589c0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:18:10.754 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:10 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:10.574+0000 7f36c4c94200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:18:11.073 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:10 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:10.811+0000 7f00589c0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:18:11.073 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:11 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:11.072+0000 7f00589c0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:18:11.116 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:10 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:10.834+0000 7f36c4c94200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:18:11.116 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:10 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:10.941+0000 7f36c4c94200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:18:11.116 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:11 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:11.024+0000 7f36c4c94200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:18:11.116 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:11 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:11.115+0000 7f36c4c94200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:18:11.354 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:11 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:11.303+0000 7f00589c0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:18:11.463 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:11 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:11.289+0000 7f36c4c94200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:18:11.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:11 smithi174 ceph-mon[30085]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:11.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:11 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:11.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:11 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:11.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:11 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:11.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:11 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:11.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:11 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:11.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:11 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:11.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:11 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:11.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:11 smithi174 ceph-mon[30085]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T22:18:11.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:11 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:18:11.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:11 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:18:11.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:11 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:11.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:11 smithi174 ceph-mon[30085]: Reconfiguring daemon mgr.y on smithi027 2024-03-11T22:18:11.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:11 smithi027 ceph-mon[25797]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:11.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:11 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:11.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:11 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:11.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:11 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:11.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:11 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:11.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:11 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:11.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:11 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:11.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:11 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:11.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:11 smithi027 ceph-mon[25797]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T22:18:11.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:11 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:18:11.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:11 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:18:11.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:11 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:11.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:11 smithi027 ceph-mon[25797]: Reconfiguring daemon mgr.y on smithi027 2024-03-11T22:18:11.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:11 smithi183 ceph-mon[30906]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:11.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:11 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:11.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:11 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:11.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:11 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:11.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:11 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:11.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:11 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:11.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:11 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:11.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:11 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:11.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:11 smithi183 ceph-mon[30906]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T22:18:11.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:11 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:18:11.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:11 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:18:11.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:11 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:11.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:11 smithi183 ceph-mon[30906]: Reconfiguring daemon mgr.y on smithi027 2024-03-11T22:18:12.353 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:12 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:12.093+0000 7f00589c0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:18:12.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:12 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/785861589' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "816905d5-722e-43eb-9a39-0af0583b6c81"}]: dispatch 2024-03-11T22:18:12.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:12 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/785861589' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "816905d5-722e-43eb-9a39-0af0583b6c81"}]': finished 2024-03-11T22:18:12.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:12 smithi174 ceph-mon[30085]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T22:18:12.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:12 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:18:12.504 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:12 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:12.239+0000 7f36c4c94200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:18:12.504 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:12 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:12.366+0000 7f36c4c94200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:18:12.760 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:12 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:12.663+0000 7f00589c0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:18:12.760 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:12 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:12.759+0000 7f00589c0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:18:12.760 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:12 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/785861589' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "816905d5-722e-43eb-9a39-0af0583b6c81"}]: dispatch 2024-03-11T22:18:12.761 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:12 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/785861589' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "816905d5-722e-43eb-9a39-0af0583b6c81"}]': finished 2024-03-11T22:18:12.761 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:12 smithi183 ceph-mon[30906]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T22:18:12.761 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:12 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:18:12.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:12 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/785861589' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "816905d5-722e-43eb-9a39-0af0583b6c81"}]: dispatch 2024-03-11T22:18:12.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:12 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/785861589' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "816905d5-722e-43eb-9a39-0af0583b6c81"}]': finished 2024-03-11T22:18:12.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:12 smithi027 ceph-mon[25797]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T22:18:12.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:12 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:18:12.974 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:12 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:12.696+0000 7f36c4c94200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:18:12.974 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:12 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:12.783+0000 7f36c4c94200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:18:12.974 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:12 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:12.973+0000 7f36c4c94200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:18:13.103 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:18:12 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:18:12.970+0000 7f00589c0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:18:13.254 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:13 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:13.150+0000 7f36c4c94200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:18:13.681 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:13 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:13.423+0000 7f36c4c94200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:18:13.682 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:13 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:13.680+0000 7f36c4c94200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:18:14.004 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:13 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:13.911+0000 7f36c4c94200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:18:14.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:13 smithi174 ceph-mon[30085]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:14.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:13 smithi174 ceph-mon[30085]: Standby manager daemon z started 2024-03-11T22:18:14.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:13 smithi174 ceph-mon[30085]: from='mgr.? 172.21.15.183:0/416443202' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T22:18:14.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:13 smithi174 ceph-mon[30085]: from='mgr.? 172.21.15.183:0/416443202' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:18:14.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:13 smithi174 ceph-mon[30085]: from='mgr.? 172.21.15.183:0/416443202' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T22:18:14.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:13 smithi174 ceph-mon[30085]: from='mgr.? 172.21.15.183:0/416443202' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:18:14.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:13 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/3097825513' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:18:14.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:13 smithi027 ceph-mon[25797]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:14.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:13 smithi027 ceph-mon[25797]: Standby manager daemon z started 2024-03-11T22:18:14.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:13 smithi027 ceph-mon[25797]: from='mgr.? 172.21.15.183:0/416443202' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T22:18:14.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:13 smithi027 ceph-mon[25797]: from='mgr.? 172.21.15.183:0/416443202' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:18:14.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:13 smithi027 ceph-mon[25797]: from='mgr.? 172.21.15.183:0/416443202' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T22:18:14.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:13 smithi027 ceph-mon[25797]: from='mgr.? 172.21.15.183:0/416443202' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:18:14.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:13 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/3097825513' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:18:14.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:13 smithi183 ceph-mon[30906]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:14.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:13 smithi183 ceph-mon[30906]: Standby manager daemon z started 2024-03-11T22:18:14.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:13 smithi183 ceph-mon[30906]: from='mgr.? 172.21.15.183:0/416443202' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T22:18:14.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:13 smithi183 ceph-mon[30906]: from='mgr.? 172.21.15.183:0/416443202' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:18:14.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:13 smithi183 ceph-mon[30906]: from='mgr.? 172.21.15.183:0/416443202' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T22:18:14.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:13 smithi183 ceph-mon[30906]: from='mgr.? 172.21.15.183:0/416443202' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:18:14.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:13 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/3097825513' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:18:15.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:14 smithi174 ceph-mon[30085]: mgrmap e14: y(active, since 79s), standbys: z 2024-03-11T22:18:15.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:14 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T22:18:15.004 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:14 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:14.687+0000 7f36c4c94200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:18:15.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:14 smithi027 ceph-mon[25797]: mgrmap e14: y(active, since 79s), standbys: z 2024-03-11T22:18:15.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:14 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T22:18:15.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:14 smithi183 ceph-mon[30906]: mgrmap e14: y(active, since 79s), standbys: z 2024-03-11T22:18:15.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:14 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T22:18:15.504 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:15 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:15.249+0000 7f36c4c94200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:18:15.504 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:15 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:15.344+0000 7f36c4c94200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:18:15.761 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:15 smithi027 ceph-mon[25797]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:15.762 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:15 smithi027 ceph-mon[25797]: Standby manager daemon x started 2024-03-11T22:18:15.762 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:15 smithi027 ceph-mon[25797]: from='mgr.? 172.21.15.174:0/1266097803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:18:15.762 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:15 smithi027 ceph-mon[25797]: from='mgr.? 172.21.15.174:0/1266097803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:18:15.762 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:15 smithi027 ceph-mon[25797]: from='mgr.? 172.21.15.174:0/1266097803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:18:15.762 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:15 smithi027 ceph-mon[25797]: from='mgr.? 172.21.15.174:0/1266097803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:18:16.004 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:18:15 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:18:15.553+0000 7f36c4c94200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:18:16.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:15 smithi174 ceph-mon[30085]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:16.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:15 smithi174 ceph-mon[30085]: Standby manager daemon x started 2024-03-11T22:18:16.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:15 smithi174 ceph-mon[30085]: from='mgr.? 172.21.15.174:0/1266097803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:18:16.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:15 smithi174 ceph-mon[30085]: from='mgr.? 172.21.15.174:0/1266097803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:18:16.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:15 smithi174 ceph-mon[30085]: from='mgr.? 172.21.15.174:0/1266097803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:18:16.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:15 smithi174 ceph-mon[30085]: from='mgr.? 172.21.15.174:0/1266097803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:18:16.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:15 smithi183 ceph-mon[30906]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:16.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:15 smithi183 ceph-mon[30906]: Standby manager daemon x started 2024-03-11T22:18:16.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:15 smithi183 ceph-mon[30906]: from='mgr.? 172.21.15.174:0/1266097803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:18:16.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:15 smithi183 ceph-mon[30906]: from='mgr.? 172.21.15.174:0/1266097803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:18:16.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:15 smithi183 ceph-mon[30906]: from='mgr.? 172.21.15.174:0/1266097803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:18:16.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:15 smithi183 ceph-mon[30906]: from='mgr.? 172.21.15.174:0/1266097803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:18:16.952 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:16 smithi027 ceph-mon[25797]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-11T22:18:16.952 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:16 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:18:17.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:16 smithi183 ceph-mon[30906]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-11T22:18:17.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:16 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:18:17.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:16 smithi174 ceph-mon[30085]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-11T22:18:17.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:16 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:18:18.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:18 smithi027 ceph-mon[25797]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:18.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:18 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:18.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:18 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:18.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:18 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:18.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:18 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:18.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:18 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:18.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:18 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:18.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:18 smithi183 ceph-mon[30906]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:18.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:18 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:18.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:18 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:18.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:18 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:18.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:18 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:18.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:18 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:18.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:18 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:18.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:18 smithi174 ceph-mon[30085]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:18.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:18 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:18.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:18 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:18.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:18 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:18.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:18 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:18.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:18 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:18.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:18 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:20.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:20 smithi027 ceph-mon[25797]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:20.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:20 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T22:18:20.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:20 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:20.330 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:20 smithi174 ceph-mon[30085]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:20.330 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:20 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T22:18:20.330 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:20 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:20.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:20 smithi183 ceph-mon[30906]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:20.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:20 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T22:18:20.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:20 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:21.302 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:21 smithi027 ceph-mon[25797]: Deploying daemon osd.0 on smithi027 2024-03-11T22:18:21.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:21 smithi183 ceph-mon[30906]: Deploying daemon osd.0 on smithi027 2024-03-11T22:18:21.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:21 smithi174 ceph-mon[30085]: Deploying daemon osd.0 on smithi027 2024-03-11T22:18:22.307 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:22 smithi027 ceph-mon[25797]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:22.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:22 smithi183 ceph-mon[30906]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:22.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:22 smithi174 ceph-mon[30085]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:24.223 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:24 smithi027 ceph-mon[25797]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:24.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:24 smithi183 ceph-mon[30906]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:24.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:24 smithi174 ceph-mon[30085]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:26.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:26 smithi027 ceph-mon[25797]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:26.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:26 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:26.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:26 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:26.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:26 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:26.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:26 smithi183 ceph-mon[30906]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:26.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:26 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:26.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:26 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:26.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:26 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:26.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:26 smithi174 ceph-mon[30085]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:26.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:26 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:26.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:26 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:26.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:26 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:28.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:28 smithi183 ceph-mon[30906]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:28.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:28 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:28.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:28 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:28.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:28 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:28.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:28 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:28.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:28 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:28.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:28 smithi174 ceph-mon[30085]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:28.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:28 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:28.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:28 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:28.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:28 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:28.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:28 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:28.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:28 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:28.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:28 smithi027 ceph-mon[25797]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:28.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:28 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:28.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:28 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:28.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:28 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:28.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:28 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:28.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:28 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:28.608 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 0 on host 'smithi027' 2024-03-11T22:18:29.228 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:29 smithi027 ceph-mon[25797]: from='osd.0 [v2:172.21.15.27:6802/328701664,v1:172.21.15.27:6803/328701664]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T22:18:29.228 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:29 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:29.228 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:29 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:29.228 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:29 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:29.255 DEBUG:teuthology.orchestra.run.smithi027:osd.0> sudo journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@osd.0.service 2024-03-11T22:18:29.257 INFO:tasks.cephadm:Deploying osd.1 on smithi174 with /dev/nvme4n1... 2024-03-11T22:18:29.257 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-11T22:18:29.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:29 smithi183 ceph-mon[30906]: from='osd.0 [v2:172.21.15.27:6802/328701664,v1:172.21.15.27:6803/328701664]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T22:18:29.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:29 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:29.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:29 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:29.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:29 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:29.498 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:18:29.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:29 smithi174 ceph-mon[30085]: from='osd.0 [v2:172.21.15.27:6802/328701664,v1:172.21.15.27:6803/328701664]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T22:18:29.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:29 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:29.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:29 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:29.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:29 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:30.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:30 smithi174 ceph-mon[30085]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:30.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:30 smithi174 ceph-mon[30085]: from='osd.0 [v2:172.21.15.27:6802/328701664,v1:172.21.15.27:6803/328701664]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T22:18:30.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:30 smithi174 ceph-mon[30085]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T22:18:30.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:30 smithi174 ceph-mon[30085]: from='osd.0 [v2:172.21.15.27:6802/328701664,v1:172.21.15.27:6803/328701664]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-03-11T22:18:30.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:30 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:18:30.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:30 smithi027 ceph-mon[25797]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:30.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:30 smithi027 ceph-mon[25797]: from='osd.0 [v2:172.21.15.27:6802/328701664,v1:172.21.15.27:6803/328701664]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T22:18:30.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:30 smithi027 ceph-mon[25797]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T22:18:30.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:30 smithi027 ceph-mon[25797]: from='osd.0 [v2:172.21.15.27:6802/328701664,v1:172.21.15.27:6803/328701664]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-03-11T22:18:30.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:30 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:18:30.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:30 smithi183 ceph-mon[30906]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:30.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:30 smithi183 ceph-mon[30906]: from='osd.0 [v2:172.21.15.27:6802/328701664,v1:172.21.15.27:6803/328701664]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T22:18:30.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:30 smithi183 ceph-mon[30906]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T22:18:30.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:30 smithi183 ceph-mon[30906]: from='osd.0 [v2:172.21.15.27:6802/328701664,v1:172.21.15.27:6803/328701664]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-03-11T22:18:30.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:30 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:18:31.022 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:18:30 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-0[44600]: 2024-03-11T22:18:30.574+0000 7f970a819700 -1 osd.0 0 waiting for initial osdmap 2024-03-11T22:18:31.022 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:18:30 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-0[44600]: 2024-03-11T22:18:30.577+0000 7f9702604700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:18:31.333 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/nvme4n1 2024-03-11T22:18:31.333 INFO:teuthology.orchestra.run.smithi174.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T22:18:31.333 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-11T22:18:31.334 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-03-11T22:18:31.334 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-03-11T22:18:31.334 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012898 s, 813 MB/s 2024-03-11T22:18:31.334 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-03-11T22:18:31.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:31 smithi183 ceph-mon[30906]: from='osd.0 [v2:172.21.15.27:6802/328701664,v1:172.21.15.27:6803/328701664]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-03-11T22:18:31.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:31 smithi183 ceph-mon[30906]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T22:18:31.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:31 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:18:31.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:31 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:18:31.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:31 smithi183 ceph-mon[30906]: from='osd.0 [v2:172.21.15.27:6802/328701664,v1:172.21.15.27:6803/328701664]' entity='osd.0' 2024-03-11T22:18:31.357 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:31 smithi174 ceph-mon[30085]: from='osd.0 [v2:172.21.15.27:6802/328701664,v1:172.21.15.27:6803/328701664]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-03-11T22:18:31.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:31 smithi174 ceph-mon[30085]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T22:18:31.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:31 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:18:31.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:31 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:18:31.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:31 smithi174 ceph-mon[30085]: from='osd.0 [v2:172.21.15.27:6802/328701664,v1:172.21.15.27:6803/328701664]' entity='osd.0' 2024-03-11T22:18:31.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:31 smithi027 ceph-mon[25797]: from='osd.0 [v2:172.21.15.27:6802/328701664,v1:172.21.15.27:6803/328701664]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-03-11T22:18:31.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:31 smithi027 ceph-mon[25797]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T22:18:31.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:31 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:18:31.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:31 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:18:31.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:31 smithi027 ceph-mon[25797]: from='osd.0 [v2:172.21.15.27:6802/328701664,v1:172.21.15.27:6803/328701664]' entity='osd.0' 2024-03-11T22:18:31.961 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi174:/dev/nvme4n1 2024-03-11T22:18:32.194 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:18:32.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:32 smithi174 ceph-mon[30085]: purged_snaps scrub starts 2024-03-11T22:18:32.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:32 smithi174 ceph-mon[30085]: purged_snaps scrub ok 2024-03-11T22:18:32.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:32 smithi174 ceph-mon[30085]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:32.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:32 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:18:32.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:32 smithi174 ceph-mon[30085]: osd.0 [v2:172.21.15.27:6802/328701664,v1:172.21.15.27:6803/328701664] boot 2024-03-11T22:18:32.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:32 smithi174 ceph-mon[30085]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T22:18:32.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:32 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:18:32.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:32 smithi027 ceph-mon[25797]: purged_snaps scrub starts 2024-03-11T22:18:32.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:32 smithi027 ceph-mon[25797]: purged_snaps scrub ok 2024-03-11T22:18:32.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:32 smithi027 ceph-mon[25797]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:32.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:32 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:18:32.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:32 smithi027 ceph-mon[25797]: osd.0 [v2:172.21.15.27:6802/328701664,v1:172.21.15.27:6803/328701664] boot 2024-03-11T22:18:32.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:32 smithi027 ceph-mon[25797]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T22:18:32.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:32 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:18:32.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:32 smithi183 ceph-mon[30906]: purged_snaps scrub starts 2024-03-11T22:18:32.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:32 smithi183 ceph-mon[30906]: purged_snaps scrub ok 2024-03-11T22:18:32.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:32 smithi183 ceph-mon[30906]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:32.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:32 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:18:32.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:32 smithi183 ceph-mon[30906]: osd.0 [v2:172.21.15.27:6802/328701664,v1:172.21.15.27:6803/328701664] boot 2024-03-11T22:18:32.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:32 smithi183 ceph-mon[30906]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T22:18:32.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:32 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:18:33.464 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:33 smithi174 ceph-mon[30085]: Detected new or changed devices on smithi027 2024-03-11T22:18:33.464 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:33 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:33.464 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:33 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:33.464 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:33 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:18:33.464 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:33 smithi174 ceph-mon[30085]: Adjusting osd_memory_target on smithi027 to 17088M 2024-03-11T22:18:33.464 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:33 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:33.464 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:33 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:33.465 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:33 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:33.465 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:33 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:33.465 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:33 smithi174 ceph-mon[30085]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:33.465 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:33 smithi174 ceph-mon[30085]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T22:18:33.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:33 smithi183 ceph-mon[30906]: Detected new or changed devices on smithi027 2024-03-11T22:18:33.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:33 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:33.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:33 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:33.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:33 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:18:33.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:33 smithi183 ceph-mon[30906]: Adjusting osd_memory_target on smithi027 to 17088M 2024-03-11T22:18:33.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:33 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:33.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:33 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:33.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:33 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:33.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:33 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:33.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:33 smithi183 ceph-mon[30906]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:33.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:33 smithi183 ceph-mon[30906]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T22:18:33.636 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:33 smithi027 ceph-mon[25797]: Detected new or changed devices on smithi027 2024-03-11T22:18:33.636 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:33 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:33.636 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:33 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:33.637 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:33 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:18:33.637 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:33 smithi027 ceph-mon[25797]: Adjusting osd_memory_target on smithi027 to 17088M 2024-03-11T22:18:33.637 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:33 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:33.637 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:33 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:33.637 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:33 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:33.637 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:33 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:33.637 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:33 smithi027 ceph-mon[25797]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:33.637 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:33 smithi027 ceph-mon[25797]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T22:18:34.597 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:34 smithi174 ceph-mon[30085]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:18:34.597 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:34 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:18:34.597 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:34 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:18:34.597 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:34 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:34.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:34 smithi183 ceph-mon[30906]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:18:34.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:34 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:18:34.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:34 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:18:34.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:34 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:34.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:34 smithi027 ceph-mon[25797]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:18:34.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:34 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:18:34.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:34 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:18:34.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:34 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:35.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:35 smithi174 ceph-mon[30085]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:35.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:35 smithi183 ceph-mon[30906]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:35.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:35 smithi027 ceph-mon[25797]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:37.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:37 smithi174 ceph-mon[30085]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:37.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:37 smithi174 ceph-mon[30085]: from='client.? 172.21.15.174:0/4212889869' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1f90f41-442c-4b79-8c7b-6db7e8d41609"}]: dispatch 2024-03-11T22:18:37.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:37 smithi174 ceph-mon[30085]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1f90f41-442c-4b79-8c7b-6db7e8d41609"}]: dispatch 2024-03-11T22:18:37.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:37 smithi174 ceph-mon[30085]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1f90f41-442c-4b79-8c7b-6db7e8d41609"}]': finished 2024-03-11T22:18:37.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:37 smithi174 ceph-mon[30085]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T22:18:37.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:37 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:18:37.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:37 smithi027 ceph-mon[25797]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:37.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:37 smithi027 ceph-mon[25797]: from='client.? 172.21.15.174:0/4212889869' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1f90f41-442c-4b79-8c7b-6db7e8d41609"}]: dispatch 2024-03-11T22:18:37.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:37 smithi027 ceph-mon[25797]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1f90f41-442c-4b79-8c7b-6db7e8d41609"}]: dispatch 2024-03-11T22:18:37.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:37 smithi027 ceph-mon[25797]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1f90f41-442c-4b79-8c7b-6db7e8d41609"}]': finished 2024-03-11T22:18:37.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:37 smithi027 ceph-mon[25797]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T22:18:37.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:37 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:18:37.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:37 smithi183 ceph-mon[30906]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:37.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:37 smithi183 ceph-mon[30906]: from='client.? 172.21.15.174:0/4212889869' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1f90f41-442c-4b79-8c7b-6db7e8d41609"}]: dispatch 2024-03-11T22:18:37.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:37 smithi183 ceph-mon[30906]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1f90f41-442c-4b79-8c7b-6db7e8d41609"}]: dispatch 2024-03-11T22:18:37.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:37 smithi183 ceph-mon[30906]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1f90f41-442c-4b79-8c7b-6db7e8d41609"}]': finished 2024-03-11T22:18:37.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:37 smithi183 ceph-mon[30906]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T22:18:37.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:37 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:18:38.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:38 smithi174 ceph-mon[30085]: from='client.? 172.21.15.174:0/3717398855' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:18:38.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:38 smithi027 ceph-mon[25797]: from='client.? 172.21.15.174:0/3717398855' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:18:38.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:38 smithi183 ceph-mon[30906]: from='client.? 172.21.15.174:0/3717398855' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:18:39.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:39 smithi174 ceph-mon[30085]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:39.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:39 smithi027 ceph-mon[25797]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:39.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:39 smithi183 ceph-mon[30906]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:41.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:41 smithi174 ceph-mon[30085]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:41.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:41 smithi027 ceph-mon[25797]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:41.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:41 smithi183 ceph-mon[30906]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:43.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:43 smithi174 ceph-mon[30085]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:43.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:43 smithi027 ceph-mon[25797]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:43.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:43 smithi183 ceph-mon[30906]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:44.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:44 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:18:44.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:44 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:44.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:44 smithi174 ceph-mon[30085]: Deploying daemon osd.1 on smithi174 2024-03-11T22:18:44.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:44 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:18:44.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:44 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:44.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:44 smithi027 ceph-mon[25797]: Deploying daemon osd.1 on smithi174 2024-03-11T22:18:44.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:44 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:18:44.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:44 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:44.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:44 smithi183 ceph-mon[30906]: Deploying daemon osd.1 on smithi174 2024-03-11T22:18:45.479 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:45 smithi183 ceph-mon[30906]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:45.661 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:45 smithi174 ceph-mon[30085]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:45.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:45 smithi027 ceph-mon[25797]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:47.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:47 smithi174 ceph-mon[30085]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:47.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:47 smithi027 ceph-mon[25797]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:47.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:47 smithi183 ceph-mon[30906]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:50.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:50 smithi174 ceph-mon[30085]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:50.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:50.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:50.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:50 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:50.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:50 smithi027 ceph-mon[25797]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:50.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:50 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:50.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:50 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:50.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:50 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:50.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:50 smithi183 ceph-mon[30906]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:50.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:50 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:50.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:50 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:50.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:50 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:52.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:52 smithi027 ceph-mon[25797]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:52.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:52 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:52.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:52 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:52.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:52 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:52.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:52 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:52.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:52 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:52.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:52 smithi183 ceph-mon[30906]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:52.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:52 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:52.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:52 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:52.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:52 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:52.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:52 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:52.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:52 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:52.438 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:52 smithi174 ceph-mon[30085]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:52.438 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:52 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:52.438 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:52 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:52.438 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:52 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:52.438 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:52 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:52.439 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:52 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:52.757 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 1 on host 'smithi174' 2024-03-11T22:18:53.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:53 smithi027 ceph-mon[25797]: from='osd.1 [v2:172.21.15.174:6800/373410677,v1:172.21.15.174:6801/373410677]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:18:53.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:53 smithi027 ceph-mon[25797]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:18:53.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:53 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:53.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:53 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:53.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:53 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:53.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:53 smithi183 ceph-mon[30906]: from='osd.1 [v2:172.21.15.174:6800/373410677,v1:172.21.15.174:6801/373410677]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:18:53.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:53 smithi183 ceph-mon[30906]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:18:53.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:53 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:53.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:53 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:53.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:53 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:53.367 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:53 smithi174 ceph-mon[30085]: from='osd.1 [v2:172.21.15.174:6800/373410677,v1:172.21.15.174:6801/373410677]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:18:53.367 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:53 smithi174 ceph-mon[30085]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:18:53.367 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:53 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:53.367 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:53 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:53.367 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:53 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:53.391 DEBUG:teuthology.orchestra.run.smithi174:osd.1> sudo journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@osd.1.service 2024-03-11T22:18:53.394 INFO:tasks.cephadm:Deploying osd.2 on smithi183 with /dev/nvme4n1... 2024-03-11T22:18:53.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-11T22:18:53.637 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:18:54.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:54 smithi183 ceph-mon[30906]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:54.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:54 smithi183 ceph-mon[30906]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T22:18:54.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:54 smithi183 ceph-mon[30906]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T22:18:54.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:54 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:18:54.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:54 smithi183 ceph-mon[30906]: from='osd.1 [v2:172.21.15.174:6800/373410677,v1:172.21.15.174:6801/373410677]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-11T22:18:54.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:54 smithi183 ceph-mon[30906]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-11T22:18:54.479 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:54 smithi174 ceph-mon[30085]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:54.479 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:54 smithi174 ceph-mon[30085]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T22:18:54.479 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:54 smithi174 ceph-mon[30085]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T22:18:54.479 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:54 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:18:54.479 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:54 smithi174 ceph-mon[30085]: from='osd.1 [v2:172.21.15.174:6800/373410677,v1:172.21.15.174:6801/373410677]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-11T22:18:54.479 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:54 smithi174 ceph-mon[30085]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-11T22:18:54.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:54 smithi027 ceph-mon[25797]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:54.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:54 smithi027 ceph-mon[25797]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T22:18:54.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:54 smithi027 ceph-mon[25797]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T22:18:54.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:54 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:18:54.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:54 smithi027 ceph-mon[25797]: from='osd.1 [v2:172.21.15.174:6800/373410677,v1:172.21.15.174:6801/373410677]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-11T22:18:54.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:54 smithi027 ceph-mon[25797]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-11T22:18:54.754 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:18:54 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-1[34647]: 2024-03-11T22:18:54.477+0000 7fce7fee9700 -1 osd.1 0 waiting for initial osdmap 2024-03-11T22:18:54.755 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:18:54 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-1[34647]: 2024-03-11T22:18:54.480+0000 7fce77cd4700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:18:55.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:55 smithi174 ceph-mon[30085]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-11T22:18:55.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:55 smithi174 ceph-mon[30085]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T22:18:55.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:55 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:18:55.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:55 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:18:55.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:55 smithi174 ceph-mon[30085]: from='osd.1 ' entity='osd.1' 2024-03-11T22:18:55.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:55 smithi183 ceph-mon[30906]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-11T22:18:55.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:55 smithi183 ceph-mon[30906]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T22:18:55.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:55 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:18:55.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:55 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:18:55.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:55 smithi183 ceph-mon[30906]: from='osd.1 ' entity='osd.1' 2024-03-11T22:18:55.498 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/nvme4n1 2024-03-11T22:18:55.499 INFO:teuthology.orchestra.run.smithi183.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T22:18:55.499 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-11T22:18:55.499 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-03-11T22:18:55.499 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-03-11T22:18:55.499 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126085 s, 832 MB/s 2024-03-11T22:18:55.499 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-03-11T22:18:55.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:55 smithi027 ceph-mon[25797]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-11T22:18:55.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:55 smithi027 ceph-mon[25797]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T22:18:55.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:55 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:18:55.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:55 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:18:55.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:55 smithi027 ceph-mon[25797]: from='osd.1 ' entity='osd.1' 2024-03-11T22:18:56.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:56 smithi183 ceph-mon[30906]: purged_snaps scrub starts 2024-03-11T22:18:56.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:56 smithi183 ceph-mon[30906]: purged_snaps scrub ok 2024-03-11T22:18:56.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:56 smithi183 ceph-mon[30906]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:56.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:56 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:18:56.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:56 smithi183 ceph-mon[30906]: osd.1 [v2:172.21.15.174:6800/373410677,v1:172.21.15.174:6801/373410677] boot 2024-03-11T22:18:56.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:56 smithi183 ceph-mon[30906]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T22:18:56.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:56 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:18:56.136 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:56 smithi174 ceph-mon[30085]: purged_snaps scrub starts 2024-03-11T22:18:56.136 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:56 smithi174 ceph-mon[30085]: purged_snaps scrub ok 2024-03-11T22:18:56.137 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:56 smithi174 ceph-mon[30085]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:56.137 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:56 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:18:56.137 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:56 smithi174 ceph-mon[30085]: osd.1 [v2:172.21.15.174:6800/373410677,v1:172.21.15.174:6801/373410677] boot 2024-03-11T22:18:56.137 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:56 smithi174 ceph-mon[30085]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T22:18:56.137 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:56 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:18:56.157 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi183:/dev/nvme4n1 2024-03-11T22:18:56.390 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:18:56.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:56 smithi027 ceph-mon[25797]: purged_snaps scrub starts 2024-03-11T22:18:56.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:56 smithi027 ceph-mon[25797]: purged_snaps scrub ok 2024-03-11T22:18:56.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:56 smithi027 ceph-mon[25797]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:18:56.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:56 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:18:56.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:56 smithi027 ceph-mon[25797]: osd.1 [v2:172.21.15.174:6800/373410677,v1:172.21.15.174:6801/373410677] boot 2024-03-11T22:18:56.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:56 smithi027 ceph-mon[25797]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T22:18:56.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:56 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:18:57.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:57 smithi183 ceph-mon[30906]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:18:57.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:57 smithi183 ceph-mon[30906]: Detected new or changed devices on smithi174 2024-03-11T22:18:57.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:57.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:57.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:18:57.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:57 smithi183 ceph-mon[30906]: Adjusting osd_memory_target on smithi174 to 17088M 2024-03-11T22:18:57.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:57.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:57.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:57.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:57.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:57 smithi183 ceph-mon[30906]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T22:18:58.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:57 smithi174 ceph-mon[30085]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:18:58.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:57 smithi174 ceph-mon[30085]: Detected new or changed devices on smithi174 2024-03-11T22:18:58.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:58.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:58.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:18:58.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:57 smithi174 ceph-mon[30085]: Adjusting osd_memory_target on smithi174 to 17088M 2024-03-11T22:18:58.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:58.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:58.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:58.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:58.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:57 smithi174 ceph-mon[30085]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T22:18:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:57 smithi027 ceph-mon[25797]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:18:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:57 smithi027 ceph-mon[25797]: Detected new or changed devices on smithi174 2024-03-11T22:18:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:18:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:57 smithi027 ceph-mon[25797]: Adjusting osd_memory_target on smithi174 to 17088M 2024-03-11T22:18:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:18:58.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:57 smithi027 ceph-mon[25797]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T22:18:58.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:58 smithi183 ceph-mon[30906]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:18:58.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:58 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:18:58.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:58 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:18:58.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:58 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:59.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:58 smithi174 ceph-mon[30085]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:18:59.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:58 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:18:59.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:58 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:18:59.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:58 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:59.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:58 smithi027 ceph-mon[25797]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:18:59.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:58 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:18:59.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:58 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:18:59.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:58 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:59.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:18:59 smithi183 ceph-mon[30906]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:00.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:18:59 smithi174 ceph-mon[30085]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:00.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:18:59 smithi027 ceph-mon[25797]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:01.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:01 smithi183 ceph-mon[30906]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:01.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:01 smithi183 ceph-mon[30906]: from='client.? 172.21.15.183:0/472890915' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fbdab51-4cef-460e-9286-b049c5a63c23"}]: dispatch 2024-03-11T22:19:01.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:01 smithi183 ceph-mon[30906]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fbdab51-4cef-460e-9286-b049c5a63c23"}]: dispatch 2024-03-11T22:19:01.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:01 smithi183 ceph-mon[30906]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fbdab51-4cef-460e-9286-b049c5a63c23"}]': finished 2024-03-11T22:19:01.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:01 smithi183 ceph-mon[30906]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T22:19:01.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:01 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:19:02.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:01 smithi174 ceph-mon[30085]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:02.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:01 smithi174 ceph-mon[30085]: from='client.? 172.21.15.183:0/472890915' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fbdab51-4cef-460e-9286-b049c5a63c23"}]: dispatch 2024-03-11T22:19:02.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:01 smithi174 ceph-mon[30085]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fbdab51-4cef-460e-9286-b049c5a63c23"}]: dispatch 2024-03-11T22:19:02.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:01 smithi174 ceph-mon[30085]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fbdab51-4cef-460e-9286-b049c5a63c23"}]': finished 2024-03-11T22:19:02.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:01 smithi174 ceph-mon[30085]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T22:19:02.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:01 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:19:02.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:01 smithi027 ceph-mon[25797]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:02.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:01 smithi027 ceph-mon[25797]: from='client.? 172.21.15.183:0/472890915' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fbdab51-4cef-460e-9286-b049c5a63c23"}]: dispatch 2024-03-11T22:19:02.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:01 smithi027 ceph-mon[25797]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fbdab51-4cef-460e-9286-b049c5a63c23"}]: dispatch 2024-03-11T22:19:02.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:01 smithi027 ceph-mon[25797]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fbdab51-4cef-460e-9286-b049c5a63c23"}]': finished 2024-03-11T22:19:02.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:01 smithi027 ceph-mon[25797]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T22:19:02.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:01 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:19:02.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:02 smithi183 ceph-mon[30906]: from='client.? 172.21.15.183:0/1128122544' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:19:02.923 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:02 smithi027 ceph-mon[25797]: from='client.? 172.21.15.183:0/1128122544' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:19:03.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:02 smithi174 ceph-mon[30085]: from='client.? 172.21.15.183:0/1128122544' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:19:03.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:03 smithi183 ceph-mon[30906]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:04.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:03 smithi174 ceph-mon[30085]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:04.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:03 smithi027 ceph-mon[25797]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:05.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:05 smithi183 ceph-mon[30906]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:06.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:05 smithi174 ceph-mon[30085]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:06.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:05 smithi027 ceph-mon[25797]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:07.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:07 smithi183 ceph-mon[30906]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:08.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:07 smithi174 ceph-mon[30085]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:08.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:07 smithi027 ceph-mon[25797]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:08.679 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:08 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T22:19:08.679 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:08 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:08.679 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:08 smithi183 ceph-mon[30906]: Deploying daemon osd.2 on smithi183 2024-03-11T22:19:08.940 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:08 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T22:19:08.940 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:08 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:08.940 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:08 smithi174 ceph-mon[30085]: Deploying daemon osd.2 on smithi183 2024-03-11T22:19:09.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:08 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T22:19:09.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:08 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:09.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:08 smithi027 ceph-mon[25797]: Deploying daemon osd.2 on smithi183 2024-03-11T22:19:09.853 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:09 smithi183 ceph-mon[30906]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:10.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:09 smithi174 ceph-mon[30085]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:10.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:09 smithi027 ceph-mon[25797]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:11.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:11 smithi183 ceph-mon[30906]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:12.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:11 smithi174 ceph-mon[30085]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:12.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:11 smithi027 ceph-mon[25797]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:14.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:14 smithi174 ceph-mon[30085]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:14.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:14 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:14.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:14 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:14.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:14 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:19:14.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:14 smithi027 ceph-mon[25797]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:14.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:14 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:14.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:14 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:14.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:14 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:19:14.587 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:14 smithi183 ceph-mon[30906]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:14.588 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:14 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:14.588 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:14 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:14.588 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:14 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:19:16.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:16 smithi174 ceph-mon[30085]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:16.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:16 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:16.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:16 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:16.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:16 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:16.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:16 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:19:16.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:16 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:16.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:16 smithi027 ceph-mon[25797]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:16.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:16 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:16.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:16 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:16.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:16 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:16.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:16 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:19:16.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:16 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:16.580 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:16 smithi183 ceph-mon[30906]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:16.580 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:16 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:16.580 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:16 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:16.580 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:16 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:16.580 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:16 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:19:16.580 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:16 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:16.962 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 2 on host 'smithi183' 2024-03-11T22:19:17.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:17 smithi174 ceph-mon[30085]: from='osd.2 [v2:172.21.15.183:6800/4216968028,v1:172.21.15.183:6801/4216968028]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:19:17.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:17 smithi174 ceph-mon[30085]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:19:17.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:17 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:17.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:17 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:17.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:17 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:19:17.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:17 smithi027 ceph-mon[25797]: from='osd.2 [v2:172.21.15.183:6800/4216968028,v1:172.21.15.183:6801/4216968028]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:19:17.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:17 smithi027 ceph-mon[25797]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:19:17.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:17 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:17.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:17 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:17.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:17 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:19:17.533 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:17 smithi183 ceph-mon[30906]: from='osd.2 [v2:172.21.15.183:6800/4216968028,v1:172.21.15.183:6801/4216968028]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:19:17.533 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:17 smithi183 ceph-mon[30906]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:19:17.533 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:17 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:17.534 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:17 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:17.534 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:17 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:19:17.558 DEBUG:teuthology.orchestra.run.smithi183:osd.2> sudo journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@osd.2.service 2024-03-11T22:19:17.560 INFO:tasks.cephadm:Waiting for 3 OSDs to come up... 2024-03-11T22:19:17.561 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T22:19:17.819 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:18.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:18 smithi027 ceph-mon[25797]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:18.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:18 smithi027 ceph-mon[25797]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T22:19:18.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:18 smithi027 ceph-mon[25797]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T22:19:18.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:18 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:19:18.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:18 smithi027 ceph-mon[25797]: from='osd.2 [v2:172.21.15.183:6800/4216968028,v1:172.21.15.183:6801/4216968028]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-11T22:19:18.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:18 smithi027 ceph-mon[25797]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-11T22:19:18.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:18 smithi183 ceph-mon[30906]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:18.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:18 smithi183 ceph-mon[30906]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T22:19:18.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:18 smithi183 ceph-mon[30906]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T22:19:18.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:18 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:19:18.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:18 smithi183 ceph-mon[30906]: from='osd.2 [v2:172.21.15.183:6800/4216968028,v1:172.21.15.183:6801/4216968028]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-11T22:19:18.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:18 smithi183 ceph-mon[30906]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-11T22:19:18.354 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:19:18 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-2[36147]: 2024-03-11T22:19:18.146+0000 7fa1b8730700 -1 osd.2 0 waiting for initial osdmap 2024-03-11T22:19:18.354 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:19:18 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-2[36147]: 2024-03-11T22:19:18.149+0000 7fa1b4523700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:19:18.453 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:18 smithi174 ceph-mon[30085]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:18.453 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:18 smithi174 ceph-mon[30085]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T22:19:18.453 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:18 smithi174 ceph-mon[30085]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T22:19:18.453 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:18 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:19:18.453 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:18 smithi174 ceph-mon[30085]: from='osd.2 [v2:172.21.15.183:6800/4216968028,v1:172.21.15.183:6801/4216968028]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-11T22:19:18.453 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:18 smithi174 ceph-mon[30085]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-11T22:19:19.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:19 smithi183 ceph-mon[30906]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-11T22:19:19.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:19 smithi183 ceph-mon[30906]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T22:19:19.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:19 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:19:19.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:19 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:19:19.413 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:19 smithi027 ceph-mon[25797]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-11T22:19:19.413 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:19 smithi027 ceph-mon[25797]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T22:19:19.413 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:19 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:19:19.413 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:19 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:19:19.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:19 smithi174 ceph-mon[30085]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-11T22:19:19.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:19 smithi174 ceph-mon[30085]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T22:19:19.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:19 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:19:19.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:19 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:19:19.856 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:19:20.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:20 smithi183 ceph-mon[30906]: purged_snaps scrub starts 2024-03-11T22:19:20.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:20 smithi183 ceph-mon[30906]: purged_snaps scrub ok 2024-03-11T22:19:20.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:20 smithi183 ceph-mon[30906]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:20.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:20 smithi183 ceph-mon[30906]: osd.2 [v2:172.21.15.183:6800/4216968028,v1:172.21.15.183:6801/4216968028] boot 2024-03-11T22:19:20.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:20 smithi183 ceph-mon[30906]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T22:19:20.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:20 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:19:20.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:20 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/3916686147' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T22:19:20.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:20 smithi174 ceph-mon[30085]: purged_snaps scrub starts 2024-03-11T22:19:20.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:20 smithi174 ceph-mon[30085]: purged_snaps scrub ok 2024-03-11T22:19:20.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:20 smithi174 ceph-mon[30085]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:20.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:20 smithi174 ceph-mon[30085]: osd.2 [v2:172.21.15.183:6800/4216968028,v1:172.21.15.183:6801/4216968028] boot 2024-03-11T22:19:20.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:20 smithi174 ceph-mon[30085]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T22:19:20.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:20 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:19:20.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:20 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/3916686147' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T22:19:20.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:20 smithi027 ceph-mon[25797]: purged_snaps scrub starts 2024-03-11T22:19:20.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:20 smithi027 ceph-mon[25797]: purged_snaps scrub ok 2024-03-11T22:19:20.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:20 smithi027 ceph-mon[25797]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:19:20.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:20 smithi027 ceph-mon[25797]: osd.2 [v2:172.21.15.183:6800/4216968028,v1:172.21.15.183:6801/4216968028] boot 2024-03-11T22:19:20.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:20 smithi027 ceph-mon[25797]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T22:19:20.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:20 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:19:20.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:20 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/3916686147' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T22:19:20.554 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":18,"num_osds":3,"num_up_osds":3,"osd_up_since":1710195559,"num_in_osds":3,"osd_in_since":1710195540,"num_remapped_pgs":0} 2024-03-11T22:19:20.554 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T22:19:20.813 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:21.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:21 smithi027 ceph-mon[25797]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T22:19:21.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:21 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:19:21.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:21 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:21.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:21 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:21.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:21 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:19:21.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:21 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:21.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:21 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:21.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:21 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:19:21.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:21 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:21.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:21 smithi174 ceph-mon[30085]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T22:19:21.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:21 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:19:21.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:21 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:21.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:21 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:21.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:21 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:19:21.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:21 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:21.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:21 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:21.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:21 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:19:21.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:21 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:21.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:21 smithi183 ceph-mon[30906]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T22:19:21.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:21 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:19:21.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:21 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:21.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:21 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:21.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:21 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:19:21.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:21 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:21.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:21 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:21.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:21 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:19:21.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:21 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:22.378 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:22 smithi027 ceph-mon[25797]: pgmap v82: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:19:22.378 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:22 smithi027 ceph-mon[25797]: Detected new or changed devices on smithi183 2024-03-11T22:19:22.378 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:22 smithi027 ceph-mon[25797]: Adjusting osd_memory_target on smithi183 to 17088M 2024-03-11T22:19:22.378 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:22 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T22:19:22.379 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:22 smithi027 ceph-mon[25797]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T22:19:22.379 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:22 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:19:22.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:22 smithi174 ceph-mon[30085]: pgmap v82: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:19:22.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:22 smithi174 ceph-mon[30085]: Detected new or changed devices on smithi183 2024-03-11T22:19:22.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:22 smithi174 ceph-mon[30085]: Adjusting osd_memory_target on smithi183 to 17088M 2024-03-11T22:19:22.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:22 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T22:19:22.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:22 smithi174 ceph-mon[30085]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T22:19:22.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:22 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:19:22.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:22 smithi183 ceph-mon[30906]: pgmap v82: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:19:22.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:22 smithi183 ceph-mon[30906]: Detected new or changed devices on smithi183 2024-03-11T22:19:22.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:22 smithi183 ceph-mon[30906]: Adjusting osd_memory_target on smithi183 to 17088M 2024-03-11T22:19:22.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:22 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T22:19:22.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:22 smithi183 ceph-mon[30906]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T22:19:22.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:22 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:19:22.840 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:19:22.840 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"c71f9f14-dff4-11ee-95c8-87774f69a715","created":"2024-03-11T22:15:57.827890+0000","modified":"2024-03-11T22:19:22.161960+0000","last_up_change":"2024-03-11T22:19:19.144835+0000","last_in_change":"2024-03-11T22:19:00.736249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:19:20.934298+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"816905d5-722e-43eb-9a39-0af0583b6c81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":328701664},{"type":"v1","addr":"172.21.15.27:6803","nonce":328701664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":328701664},{"type":"v1","addr":"172.21.15.27:6805","nonce":328701664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":328701664},{"type":"v1","addr":"172.21.15.27:6809","nonce":328701664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":328701664},{"type":"v1","addr":"172.21.15.27:6807","nonce":328701664}]},"public_addr":"172.21.15.27:6803/328701664","cluster_addr":"172.21.15.27:6805/328701664","heartbeat_back_addr":"172.21.15.27:6809/328701664","heartbeat_front_addr":"172.21.15.27:6807/328701664","state":["exists","up"]},{"osd":1,"uuid":"d1f90f41-442c-4b79-8c7b-6db7e8d41609","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":373410677},{"type":"v1","addr":"172.21.15.174:6801","nonce":373410677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":373410677},{"type":"v1","addr":"172.21.15.174:6803","nonce":373410677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":373410677},{"type":"v1","addr":"172.21.15.174:6807","nonce":373410677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":373410677},{"type":"v1","addr":"172.21.15.174:6805","nonce":373410677}]},"public_addr":"172.21.15.174:6801/373410677","cluster_addr":"172.21.15.174:6803/373410677","heartbeat_back_addr":"172.21.15.174:6807/373410677","heartbeat_front_addr":"172.21.15.174:6805/373410677","state":["exists","up"]},{"osd":2,"uuid":"1fbdab51-4cef-460e-9286-b049c5a63c23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":4216968028},{"type":"v1","addr":"172.21.15.183:6801","nonce":4216968028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":4216968028},{"type":"v1","addr":"172.21.15.183:6803","nonce":4216968028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":4216968028},{"type":"v1","addr":"172.21.15.183:6807","nonce":4216968028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":4216968028},{"type":"v1","addr":"172.21.15.183:6805","nonce":4216968028}]},"public_addr":"172.21.15.183:6801/4216968028","cluster_addr":"172.21.15.183:6803/4216968028","heartbeat_back_addr":"172.21.15.183:6807/4216968028","heartbeat_front_addr":"172.21.15.183:6805/4216968028","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:18:29.173381+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:18:53.145160+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:19:17.272070+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/990800086":"2024-03-12T22:16:54.388610+0000","172.21.15.27:6801/827022794":"2024-03-12T22:16:54.388610+0000","172.21.15.27:0/2902087996":"2024-03-12T22:16:54.388610+0000","172.21.15.27:0/2389474342":"2024-03-12T22:16:26.285486+0000","172.21.15.27:6800/827022794":"2024-03-12T22:16:54.388610+0000","172.21.15.27:0/1407031812":"2024-03-12T22:16:26.285486+0000","172.21.15.27:6800/1706259253":"2024-03-12T22:16:26.285486+0000","172.21.15.27:6801/1706259253":"2024-03-12T22:16:26.285486+0000","172.21.15.27:0/814977505":"2024-03-12T22:16:26.285486+0000","172.21.15.27:0/2606167595":"2024-03-12T22:16:54.388610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:19:23.022 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:19:22 smithi027 sudo[50113]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-11T22:19:23.022 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:19:22 smithi027 sudo[50113]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:19:23.022 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:19:22 smithi027 sudo[50113]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:19:23.022 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:19:22 smithi027 sudo[50113]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:19:23.426 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:23 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:19:23.426 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:23 smithi174 ceph-mon[30085]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T22:19:23.426 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:23 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/2064726567' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:19:23.456 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:23 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:19:23.457 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:23 smithi027 ceph-mon[25797]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T22:19:23.457 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:23 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/2064726567' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:19:23.457 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:19:23 smithi027 sudo[50137]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-11T22:19:23.457 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:19:23 smithi027 sudo[50137]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:19:23.457 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:19:23 smithi027 sudo[50137]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:19:23.457 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:19:23 smithi027 sudo[50137]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:19:23.484 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T22:19:20.934298+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T22:19:23.484 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-11T22:19:23.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:23 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:19:23.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:23 smithi183 ceph-mon[30906]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T22:19:23.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:23 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/2064726567' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:19:23.677 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:19:23 smithi174 sudo[39544]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-11T22:19:23.677 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:19:23 smithi174 sudo[39544]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:19:23.677 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:19:23 smithi174 sudo[39544]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:19:23.678 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:19:23 smithi174 sudo[39544]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:19:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:24.004 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:19:23 smithi174 sudo[39547]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-11T22:19:24.004 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:19:23 smithi174 sudo[39547]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:19:24.004 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:19:23 smithi174 sudo[39547]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:19:24.005 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:19:23 smithi174 sudo[39547]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:19:24.195 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:24 smithi183 ceph-mon[30906]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:19:24.195 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:24 smithi183 ceph-mon[30906]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T22:19:24.195 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:24 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:24.195 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:24 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:24.195 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:19:23 smithi183 sudo[41726]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-11T22:19:24.195 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:19:23 smithi183 sudo[41726]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:19:24.195 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:19:23 smithi183 sudo[41726]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:19:24.195 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:19:23 smithi183 sudo[41726]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:19:24.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:24 smithi174 ceph-mon[30085]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:19:24.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:24 smithi174 ceph-mon[30085]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T22:19:24.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:24 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:24.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:24 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:24.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:24 smithi027 ceph-mon[25797]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:19:24.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:24 smithi027 ceph-mon[25797]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T22:19:24.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:24 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:24.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:24 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:24.603 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:19:24 smithi183 sudo[41729]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-11T22:19:24.603 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:19:24 smithi183 sudo[41729]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:19:24.603 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:19:24 smithi183 sudo[41729]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:19:24.603 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:19:24 smithi183 sudo[41729]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:19:24.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:24 smithi027 sudo[50258]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:19:24.846 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:24 smithi027 sudo[50258]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:19:24.846 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:24 smithi027 sudo[50258]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:19:24.846 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:24 smithi027 sudo[50258]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:19:25.391 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:25 smithi027 sudo[50282]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:19:25.391 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:25 smithi027 sudo[50282]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:19:25.391 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:25 smithi027 sudo[50282]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:19:25.391 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:25 smithi027 sudo[50282]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:19:25.626 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-11T22:19:25.650 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:25 smithi183 sudo[41732]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:19:25.650 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:25 smithi183 sudo[41732]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:19:25.650 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:25 smithi183 sudo[41732]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:19:25.650 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:25 smithi183 sudo[41732]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:19:25.917 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:25 smithi174 ceph-mon[30085]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-11T22:19:25.917 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:25 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:25.917 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:25 smithi174 ceph-mon[30085]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:19:25.917 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:25 smithi174 ceph-mon[30085]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:19:25.917 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:25 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:19:25.917 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:25 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:19:25.917 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:25 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:19:25.917 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:25 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:25.917 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:25 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:19:25.917 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:25 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:19:25.918 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:25 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:19:25.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:25 smithi027 ceph-mon[25797]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-11T22:19:25.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:25 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:25.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:25 smithi027 ceph-mon[25797]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:19:25.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:25 smithi027 ceph-mon[25797]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:19:25.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:25 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:19:25.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:25 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:19:25.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:25 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:19:25.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:25 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:25.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:25 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:19:25.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:25 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:19:25.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:25 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:19:26.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:25 smithi183 sudo[41735]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:19:26.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:25 smithi183 sudo[41735]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:19:26.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:25 smithi183 sudo[41735]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:19:26.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:25 smithi183 sudo[41735]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:19:26.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:25 smithi183 ceph-mon[30906]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-11T22:19:26.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:25 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:26.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:25 smithi183 ceph-mon[30906]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:19:26.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:25 smithi183 ceph-mon[30906]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:19:26.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:25 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:19:26.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:25 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:19:26.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:25 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:19:26.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:25 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:26.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:25 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:19:26.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:25 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:19:26.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:25 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:19:26.170 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:25 smithi174 sudo[39550]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:19:26.170 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:25 smithi174 sudo[39550]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:19:26.170 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:25 smithi174 sudo[39550]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:19:26.170 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:25 smithi174 sudo[39550]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:19:26.264 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi027 2024-03-11T22:19:26.264 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi027=foo.a' 2024-03-11T22:19:26.498 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:19:26.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:26 smithi174 sudo[39553]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:19:26.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:26 smithi174 sudo[39553]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:19:26.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:26 smithi174 sudo[39553]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:19:26.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:26 smithi174 sudo[39553]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:19:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:26 smithi183 ceph-mon[30906]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:19:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:26 smithi183 ceph-mon[30906]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:19:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:26 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/1132027633' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T22:19:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:26 smithi183 ceph-mon[30906]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:19:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:26 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:26 smithi183 ceph-mon[30906]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:19:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:26 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:19:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:26 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:19:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:26 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:19:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:26 smithi183 ceph-mon[30906]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:19:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:26 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:27.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:26 smithi174 ceph-mon[30085]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:19:27.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:26 smithi174 ceph-mon[30085]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:19:27.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:26 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/1132027633' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T22:19:27.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:26 smithi174 ceph-mon[30085]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:19:27.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:26 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:27.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:26 smithi174 ceph-mon[30085]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:19:27.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:26 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:19:27.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:26 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:19:27.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:26 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:19:27.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:26 smithi174 ceph-mon[30085]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:19:27.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:26 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:27.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:26 smithi027 ceph-mon[25797]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:19:27.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:26 smithi027 ceph-mon[25797]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:19:27.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:26 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/1132027633' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T22:19:27.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:26 smithi027 ceph-mon[25797]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:19:27.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:26 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:27.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:26 smithi027 ceph-mon[25797]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:19:27.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:26 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:19:27.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:26 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:19:27.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:26 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:19:27.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:26 smithi027 ceph-mon[25797]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:19:27.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:26 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:28.150 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled rgw.foo.a update... 2024-03-11T22:19:28.499 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:28 smithi183 ceph-mon[30906]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:19:28.499 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:28 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:28.499 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:28 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:19:28.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:28 smithi174 ceph-mon[30085]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:19:28.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:28 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:28.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:28 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:19:28.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:28 smithi027 ceph-mon[25797]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:19:28.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:28 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:28.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:28 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:19:28.783 DEBUG:teuthology.orchestra.run.smithi027:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@rgw.foo.a.service 2024-03-11T22:19:28.786 INFO:tasks.cephadm:Adding prometheus.a on smithi174 2024-03-11T22:19:28.787 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph orch apply prometheus '1;smithi174=a' 2024-03-11T22:19:29.015 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:19:29.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:29 smithi183 ceph-mon[30906]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi027=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:19:29.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:29 smithi183 ceph-mon[30906]: Saving service rgw.foo.a spec with placement smithi027=foo.a;count:1 2024-03-11T22:19:29.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:29 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:29.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:29 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:19:29.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:29 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:29.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:29 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T22:19:29.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:29 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T22:19:29.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:29 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:29.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:29 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:29.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:29 smithi027 ceph-mon[25797]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi027=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:19:29.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:29 smithi027 ceph-mon[25797]: Saving service rgw.foo.a spec with placement smithi027=foo.a;count:1 2024-03-11T22:19:29.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:29 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:29.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:29 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:19:29.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:29 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:29.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:29 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T22:19:29.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:29 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T22:19:29.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:29 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:29.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:29 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:29.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:29 smithi174 ceph-mon[30085]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi027=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:19:29.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:29 smithi174 ceph-mon[30085]: Saving service rgw.foo.a spec with placement smithi027=foo.a;count:1 2024-03-11T22:19:29.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:29 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:29.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:29 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:19:29.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:29 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:29.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:29 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T22:19:29.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:29 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T22:19:29.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:29 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:29.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:29 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:30.341 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:30 smithi027 ceph-mon[25797]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:19:30.341 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:30 smithi027 ceph-mon[25797]: Deploying daemon rgw.foo.a on smithi027 2024-03-11T22:19:30.383 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:30 smithi183 ceph-mon[30906]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:19:30.383 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:30 smithi183 ceph-mon[30906]: Deploying daemon rgw.foo.a on smithi027 2024-03-11T22:19:30.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:30 smithi174 ceph-mon[30085]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:19:30.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:30 smithi174 ceph-mon[30085]: Deploying daemon rgw.foo.a on smithi027 2024-03-11T22:19:30.818 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled prometheus update... 2024-03-11T22:19:31.022 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Mar 11 22:19:30 smithi027 systemd[1]: Starting Ceph rgw.foo.a for c71f9f14-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:19:31.388 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Mar 11 22:19:31 smithi027 podman[50664]: 2024-03-11 22:19:30.945699651 +0000 UTC m=+0.018702016 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:19:31.388 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Mar 11 22:19:31 smithi027 podman[50664]: 2024-03-11 22:19:31.050681641 +0000 UTC m=+0.123684035 container create 9cb6f13adf41e941e47b97cde65484137c7a1f4777a10501fff285f0ece0734d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-rgw-foo-a, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-11T22:19:31.388 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Mar 11 22:19:31 smithi027 podman[50664]: 2024-03-11 22:19:31.286655217 +0000 UTC m=+0.359657571 container init 9cb6f13adf41e941e47b97cde65484137c7a1f4777a10501fff285f0ece0734d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-rgw-foo-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-11T22:19:31.388 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Mar 11 22:19:31 smithi027 podman[50664]: 2024-03-11 22:19:31.292457531 +0000 UTC m=+0.365459880 container start 9cb6f13adf41e941e47b97cde65484137c7a1f4777a10501fff285f0ece0734d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-03-11T22:19:31.388 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Mar 11 22:19:31 smithi027 bash[50664]: 9cb6f13adf41e941e47b97cde65484137c7a1f4777a10501fff285f0ece0734d 2024-03-11T22:19:31.464 DEBUG:teuthology.orchestra.run.smithi174:prometheus.a> sudo journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@prometheus.a.service 2024-03-11T22:19:31.467 INFO:tasks.cephadm:Adding node-exporter.a on smithi027 2024-03-11T22:19:31.467 INFO:tasks.cephadm:Adding node-exporter.b on smithi174 2024-03-11T22:19:31.467 INFO:tasks.cephadm:Adding node-exporter.c on smithi183 2024-03-11T22:19:31.467 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph orch apply node-exporter '3;smithi027=a;smithi174=b;smithi183=c' 2024-03-11T22:19:31.697 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:19:31.772 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Mar 11 22:19:31 smithi027 systemd[1]: Started Ceph rgw.foo.a for c71f9f14-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:19:32.063 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:31 smithi183 ceph-mon[30906]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:19:32.063 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:31 smithi183 ceph-mon[30906]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi174=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:19:32.063 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:31 smithi183 ceph-mon[30906]: Saving service prometheus spec with placement smithi174=a;count:1 2024-03-11T22:19:32.063 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:31 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:32.063 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:31 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:32.063 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:31 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:32.063 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:31 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:32.064 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:31 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:32.064 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:31 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:32.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:31 smithi174 ceph-mon[30085]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:19:32.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:31 smithi174 ceph-mon[30085]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi174=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:19:32.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:31 smithi174 ceph-mon[30085]: Saving service prometheus spec with placement smithi174=a;count:1 2024-03-11T22:19:32.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:31 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:32.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:31 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:32.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:31 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:32.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:31 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:32.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:31 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:32.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:31 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:32.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:31 smithi027 ceph-mon[25797]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:19:32.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:31 smithi027 ceph-mon[25797]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi174=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:19:32.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:31 smithi027 ceph-mon[25797]: Saving service prometheus spec with placement smithi174=a;count:1 2024-03-11T22:19:32.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:31 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:32.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:31 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:32.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:31 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:32.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:31 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:32.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:31 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:32.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:31 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:33.035 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:32 smithi183 ceph-mon[30906]: Saving service rgw.foo.a spec with placement smithi027=foo.a;count:1 2024-03-11T22:19:33.035 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:32 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:19:33.035 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:32 smithi183 ceph-mon[30906]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T22:19:33.035 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:32 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/2495716226' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T22:19:33.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:32 smithi174 ceph-mon[30085]: Saving service rgw.foo.a spec with placement smithi027=foo.a;count:1 2024-03-11T22:19:33.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:32 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:19:33.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:32 smithi174 ceph-mon[30085]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T22:19:33.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:32 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/2495716226' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T22:19:33.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:32 smithi027 ceph-mon[25797]: Saving service rgw.foo.a spec with placement smithi027=foo.a;count:1 2024-03-11T22:19:33.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:32 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:19:33.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:32 smithi027 ceph-mon[25797]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T22:19:33.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:32 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/2495716226' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T22:19:33.737 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled node-exporter update... 2024-03-11T22:19:34.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:33 smithi027 ceph-mon[25797]: pgmap v92: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-11T22:19:34.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:33 smithi027 ceph-mon[25797]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:19:34.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:33 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/2495716226' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T22:19:34.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:33 smithi027 ceph-mon[25797]: osdmap e24: 3 total, 3 up, 3 in 2024-03-11T22:19:34.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:33 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:34.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:33 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:34.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:33 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:34.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:33 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:34.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:33 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:19:34.039 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:33 smithi183 ceph-mon[30906]: pgmap v92: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-11T22:19:34.039 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:33 smithi183 ceph-mon[30906]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:19:34.039 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:33 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/2495716226' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T22:19:34.039 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:33 smithi183 ceph-mon[30906]: osdmap e24: 3 total, 3 up, 3 in 2024-03-11T22:19:34.039 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:33 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:34.039 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:33 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:34.039 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:33 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:34.039 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:33 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:34.039 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:33 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:19:34.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:33 smithi174 ceph-mon[30085]: pgmap v92: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-11T22:19:34.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:33 smithi174 ceph-mon[30085]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:19:34.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:33 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/2495716226' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T22:19:34.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:33 smithi174 ceph-mon[30085]: osdmap e24: 3 total, 3 up, 3 in 2024-03-11T22:19:34.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:33 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:34.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:33 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:34.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:33 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:34.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:33 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:34.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:33 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:19:34.365 DEBUG:teuthology.orchestra.run.smithi027:node-exporter.a> sudo journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@node-exporter.a.service 2024-03-11T22:19:34.368 DEBUG:teuthology.orchestra.run.smithi174:node-exporter.b> sudo journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@node-exporter.b.service 2024-03-11T22:19:34.371 DEBUG:teuthology.orchestra.run.smithi183:node-exporter.c> sudo journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@node-exporter.c.service 2024-03-11T22:19:34.374 INFO:tasks.cephadm:Adding alertmanager.a on smithi027 2024-03-11T22:19:34.374 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph orch apply alertmanager '1;smithi027=a' 2024-03-11T22:19:34.643 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:19:35.014 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:34 smithi183 ceph-mon[30906]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi027=a;smithi174=b;smithi183=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:19:35.014 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:34 smithi183 ceph-mon[30906]: Saving service node-exporter spec with placement smithi027=a;smithi174=b;smithi183=c;count:3 2024-03-11T22:19:35.015 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:34 smithi183 ceph-mon[30906]: Checking dashboard <-> RGW credentials 2024-03-11T22:19:35.015 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:34 smithi183 ceph-mon[30906]: osdmap e25: 3 total, 3 up, 3 in 2024-03-11T22:19:35.015 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:34 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:19:35.015 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:34 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:19:35.015 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:34 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:35.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:34 smithi174 ceph-mon[30085]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi027=a;smithi174=b;smithi183=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:19:35.254 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:34 smithi174 ceph-mon[30085]: Saving service node-exporter spec with placement smithi027=a;smithi174=b;smithi183=c;count:3 2024-03-11T22:19:35.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:34 smithi174 ceph-mon[30085]: Checking dashboard <-> RGW credentials 2024-03-11T22:19:35.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:34 smithi174 ceph-mon[30085]: osdmap e25: 3 total, 3 up, 3 in 2024-03-11T22:19:35.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:34 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:19:35.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:34 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:19:35.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:34 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:35.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:34 smithi027 ceph-mon[25797]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi027=a;smithi174=b;smithi183=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:19:35.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:34 smithi027 ceph-mon[25797]: Saving service node-exporter spec with placement smithi027=a;smithi174=b;smithi183=c;count:3 2024-03-11T22:19:35.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:34 smithi027 ceph-mon[25797]: Checking dashboard <-> RGW credentials 2024-03-11T22:19:35.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:34 smithi027 ceph-mon[25797]: osdmap e25: 3 total, 3 up, 3 in 2024-03-11T22:19:35.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:34 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:19:35.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:34 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:19:35.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:34 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:36.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:36 smithi183 ceph-mon[30906]: pgmap v95: 65 pgs: 64 unknown, 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-11T22:19:36.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:36 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:19:36.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:36 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:19:36.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:36 smithi183 ceph-mon[30906]: osdmap e26: 3 total, 3 up, 3 in 2024-03-11T22:19:36.378 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled alertmanager update... 2024-03-11T22:19:36.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:36 smithi174 ceph-mon[30085]: pgmap v95: 65 pgs: 64 unknown, 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-11T22:19:36.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:36 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:19:36.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:36 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:19:36.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:36 smithi174 ceph-mon[30085]: osdmap e26: 3 total, 3 up, 3 in 2024-03-11T22:19:36.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:36 smithi027 ceph-mon[25797]: pgmap v95: 65 pgs: 64 unknown, 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-11T22:19:36.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:36 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:19:36.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:36 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:19:36.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:36 smithi027 ceph-mon[25797]: osdmap e26: 3 total, 3 up, 3 in 2024-03-11T22:19:36.985 DEBUG:teuthology.orchestra.run.smithi027:alertmanager.a> sudo journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@alertmanager.a.service 2024-03-11T22:19:36.988 INFO:tasks.cephadm:Adding grafana.a on smithi174 2024-03-11T22:19:36.989 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph orch apply grafana '1;smithi174=a' 2024-03-11T22:19:37.223 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:19:37.595 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:37 smithi183 ceph-mon[30906]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi027=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:19:37.595 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:37 smithi183 ceph-mon[30906]: Saving service alertmanager spec with placement smithi027=a;count:1 2024-03-11T22:19:37.595 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:37 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:37.595 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:37 smithi183 ceph-mon[30906]: osdmap e27: 3 total, 3 up, 3 in 2024-03-11T22:19:37.596 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:37 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:19:37.596 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:37 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:19:37.596 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:37 smithi183 ceph-mon[30906]: pgmap v98: 97 pgs: 9 creating+peering, 36 active+clean, 52 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T22:19:37.656 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:37 smithi174 ceph-mon[30085]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi027=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:19:37.656 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:37 smithi174 ceph-mon[30085]: Saving service alertmanager spec with placement smithi027=a;count:1 2024-03-11T22:19:37.656 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:37 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:37.656 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:37 smithi174 ceph-mon[30085]: osdmap e27: 3 total, 3 up, 3 in 2024-03-11T22:19:37.656 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:37 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:19:37.656 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:37 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:19:37.657 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:37 smithi174 ceph-mon[30085]: pgmap v98: 97 pgs: 9 creating+peering, 36 active+clean, 52 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T22:19:37.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:37 smithi027 ceph-mon[25797]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi027=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:19:37.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:37 smithi027 ceph-mon[25797]: Saving service alertmanager spec with placement smithi027=a;count:1 2024-03-11T22:19:37.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:37 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:37.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:37 smithi027 ceph-mon[25797]: osdmap e27: 3 total, 3 up, 3 in 2024-03-11T22:19:37.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:37 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:19:37.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:37 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:19:37.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:37 smithi027 ceph-mon[25797]: pgmap v98: 97 pgs: 9 creating+peering, 36 active+clean, 52 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T22:19:38.585 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:38 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:19:38.585 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:38 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:19:38.585 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:38 smithi183 ceph-mon[30906]: osdmap e28: 3 total, 3 up, 3 in 2024-03-11T22:19:38.672 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:38 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:19:38.672 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:38 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:19:38.672 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:38 smithi174 ceph-mon[30085]: osdmap e28: 3 total, 3 up, 3 in 2024-03-11T22:19:38.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:38 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:19:38.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:38 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:19:38.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:38 smithi027 ceph-mon[25797]: osdmap e28: 3 total, 3 up, 3 in 2024-03-11T22:19:38.906 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled grafana update... 2024-03-11T22:19:39.471 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:39 smithi183 ceph-mon[30906]: osdmap e29: 3 total, 3 up, 3 in 2024-03-11T22:19:39.471 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:39 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:19:39.471 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:39 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:19:39.471 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:39 smithi183 ceph-mon[30906]: pgmap v101: 129 pgs: 20 creating+peering, 56 active+clean, 53 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-11T22:19:39.471 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:39 smithi183 ceph-mon[30906]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi174=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:19:39.471 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:39 smithi183 ceph-mon[30906]: Saving service grafana spec with placement smithi174=a;count:1 2024-03-11T22:19:39.471 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:39 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:39.497 DEBUG:teuthology.orchestra.run.smithi174:grafana.a> sudo journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@grafana.a.service 2024-03-11T22:19:39.500 INFO:tasks.cephadm:Setting up client nodes... 2024-03-11T22:19:39.500 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T22:19:39.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:39 smithi174 ceph-mon[30085]: osdmap e29: 3 total, 3 up, 3 in 2024-03-11T22:19:39.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:39 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:19:39.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:39 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:19:39.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:39 smithi174 ceph-mon[30085]: pgmap v101: 129 pgs: 20 creating+peering, 56 active+clean, 53 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-11T22:19:39.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:39 smithi174 ceph-mon[30085]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi174=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:19:39.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:39 smithi174 ceph-mon[30085]: Saving service grafana spec with placement smithi174=a;count:1 2024-03-11T22:19:39.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:39 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:39.760 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:39.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:39 smithi027 ceph-mon[25797]: osdmap e29: 3 total, 3 up, 3 in 2024-03-11T22:19:39.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:39 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:19:39.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:39 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:19:39.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:39 smithi027 ceph-mon[25797]: pgmap v101: 129 pgs: 20 creating+peering, 56 active+clean, 53 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-11T22:19:39.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:39 smithi027 ceph-mon[25797]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi174=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:19:39.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:39 smithi027 ceph-mon[25797]: Saving service grafana spec with placement smithi174=a;count:1 2024-03-11T22:19:39.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:39 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:40.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:40 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:19:40.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:40 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:19:40.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:40 smithi174 ceph-mon[30085]: osdmap e30: 3 total, 3 up, 3 in 2024-03-11T22:19:40.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:40 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:19:40.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:40 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:19:40.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:40 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:19:40.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:40 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:19:40.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:40 smithi027 ceph-mon[25797]: osdmap e30: 3 total, 3 up, 3 in 2024-03-11T22:19:40.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:40 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:19:40.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:40 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:19:40.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:40 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:19:40.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:40 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:19:40.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:40 smithi183 ceph-mon[30906]: osdmap e30: 3 total, 3 up, 3 in 2024-03-11T22:19:40.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:40 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:19:40.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:40 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:19:41.662 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-03-11T22:19:41.662 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQB9g+9lPGZPJxAAdYEMA2ixs5rPKx+oqCSoJQ== 2024-03-11T22:19:41.674 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:41 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:19:41.674 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:41 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:19:41.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:41 smithi027 ceph-mon[25797]: osdmap e31: 3 total, 3 up, 3 in 2024-03-11T22:19:41.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:41 smithi027 ceph-mon[25797]: pgmap v104: 129 pgs: 11 creating+peering, 86 active+clean, 32 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-11T22:19:41.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:41 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:41.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:41 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:41.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:41 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:41.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:41 smithi027 ceph-mon[25797]: Deploying daemon prometheus.a on smithi174 2024-03-11T22:19:41.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:41 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:19:41.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:41 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:19:41.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:41 smithi174 ceph-mon[30085]: osdmap e31: 3 total, 3 up, 3 in 2024-03-11T22:19:41.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:41 smithi174 ceph-mon[30085]: pgmap v104: 129 pgs: 11 creating+peering, 86 active+clean, 32 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-11T22:19:41.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:41 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:41.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:41 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:41.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:41 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:41.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:41 smithi174 ceph-mon[30085]: Deploying daemon prometheus.a on smithi174 2024-03-11T22:19:41.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:41 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/2500020338' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:19:41.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:41 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/4086278763' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:19:41.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:41 smithi183 ceph-mon[30906]: osdmap e31: 3 total, 3 up, 3 in 2024-03-11T22:19:41.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:41 smithi183 ceph-mon[30906]: pgmap v104: 129 pgs: 11 creating+peering, 86 active+clean, 32 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-11T22:19:41.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:41.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:41.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:41 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:41.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:41 smithi183 ceph-mon[30906]: Deploying daemon prometheus.a on smithi174 2024-03-11T22:19:42.246 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-11T22:19:42.246 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-11T22:19:42.246 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-11T22:19:42.296 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T22:19:42.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:42 smithi027 ceph-mon[25797]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:19:42.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:42 smithi027 ceph-mon[25797]: Cluster is now healthy 2024-03-11T22:19:42.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:42 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/2068464504' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:19:42.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:42 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/2068464504' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:19:42.529 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:19:42.652 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:42 smithi174 ceph-mon[30085]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:19:42.652 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:42 smithi174 ceph-mon[30085]: Cluster is now healthy 2024-03-11T22:19:42.652 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:42 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/2068464504' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:19:42.652 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:42 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/2068464504' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:19:42.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:42 smithi183 ceph-mon[30906]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:19:42.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:42 smithi183 ceph-mon[30906]: Cluster is now healthy 2024-03-11T22:19:42.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:42 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/2068464504' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:19:42.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:42 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/2068464504' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:19:43.986 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:43 smithi027 ceph-mon[25797]: pgmap v105: 129 pgs: 129 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 67 KiB/s rd, 4.8 KiB/s wr, 162 op/s 2024-03-11T22:19:44.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:43 smithi174 ceph-mon[30085]: pgmap v105: 129 pgs: 129 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 67 KiB/s rd, 4.8 KiB/s wr, 162 op/s 2024-03-11T22:19:44.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:43 smithi183 ceph-mon[30906]: pgmap v105: 129 pgs: 129 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 67 KiB/s rd, 4.8 KiB/s wr, 162 op/s 2024-03-11T22:19:45.292 INFO:teuthology.orchestra.run.smithi174.stdout:[client.1] 2024-03-11T22:19:45.292 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQCBg+9lKkEhERAAh6akhdqGw+OXZ7x8LKUFdg== 2024-03-11T22:19:46.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:45 smithi174 ceph-mon[30085]: pgmap v106: 129 pgs: 129 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 56 KiB/s rd, 4.0 KiB/s wr, 135 op/s 2024-03-11T22:19:46.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:45 smithi174 ceph-mon[30085]: from='client.? 172.21.15.174:0/829900452' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:19:46.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:45 smithi174 ceph-mon[30085]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:19:46.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:45 smithi174 ceph-mon[30085]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:19:46.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:45 smithi027 ceph-mon[25797]: pgmap v106: 129 pgs: 129 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 56 KiB/s rd, 4.0 KiB/s wr, 135 op/s 2024-03-11T22:19:46.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:45 smithi027 ceph-mon[25797]: from='client.? 172.21.15.174:0/829900452' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:19:46.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:45 smithi027 ceph-mon[25797]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:19:46.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:45 smithi027 ceph-mon[25797]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:19:46.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:45 smithi183 ceph-mon[30906]: pgmap v106: 129 pgs: 129 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 56 KiB/s rd, 4.0 KiB/s wr, 135 op/s 2024-03-11T22:19:46.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:45 smithi183 ceph-mon[30906]: from='client.? 172.21.15.174:0/829900452' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:19:46.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:45 smithi183 ceph-mon[30906]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:19:46.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:45 smithi183 ceph-mon[30906]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:19:46.245 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-11T22:19:46.245 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-11T22:19:46.245 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-11T22:19:46.288 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T22:19:46.521 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:19:47.902 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:47 smithi183 ceph-mon[30906]: pgmap v107: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-03-11T22:19:48.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:47 smithi174 ceph-mon[30085]: pgmap v107: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-03-11T22:19:48.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:47 smithi027 ceph-mon[25797]: pgmap v107: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-03-11T22:19:48.422 INFO:teuthology.orchestra.run.smithi183.stdout:[client.2] 2024-03-11T22:19:48.422 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQCEg+9lQArnGBAAPO7kjcXoBLD03CfN9af5yQ== 2024-03-11T22:19:49.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:48 smithi174 ceph-mon[30085]: from='client.? 172.21.15.183:0/3353256472' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:19:49.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:48 smithi174 ceph-mon[30085]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:19:49.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:48 smithi174 ceph-mon[30085]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:19:49.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:48 smithi027 ceph-mon[25797]: from='client.? 172.21.15.183:0/3353256472' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:19:49.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:48 smithi027 ceph-mon[25797]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:19:49.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:48 smithi027 ceph-mon[25797]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:19:49.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:48 smithi183 ceph-mon[30906]: from='client.? 172.21.15.183:0/3353256472' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:19:49.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:48 smithi183 ceph-mon[30906]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:19:49.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:48 smithi183 ceph-mon[30906]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:19:49.198 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-11T22:19:49.198 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-03-11T22:19:49.198 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-11T22:19:49.243 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-11T22:19:49.244 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-11T22:19:49.244 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-11T22:19:49.503 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:49.930 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:49 smithi027 ceph-mon[25797]: pgmap v108: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 52 KiB/s rd, 4.0 KiB/s wr, 125 op/s 2024-03-11T22:19:50.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:49 smithi174 ceph-mon[30085]: pgmap v108: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 52 KiB/s rd, 4.0 KiB/s wr, 125 op/s 2024-03-11T22:19:50.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:49 smithi183 ceph-mon[30906]: pgmap v108: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 52 KiB/s rd, 4.0 KiB/s wr, 125 op/s 2024-03-11T22:19:51.405 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:19:51.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:51 smithi027 ceph-mon[25797]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.6 KiB/s wr, 112 op/s 2024-03-11T22:19:51.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:51 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/1225213171' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T22:19:51.993 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":4164125349},{"type":"v1","addr":"172.21.15.27:6801","nonce":4164125349}]},"active_addr":"172.21.15.27:6801/4164125349","active_change":"2024-03-11T22:16:54.388719+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.27:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":235672847}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":1667873686}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":3082796003}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":4267752929}]}]} 2024-03-11T22:19:51.999 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-11T22:19:51.999 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T22:19:51.999 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T22:19:52.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:51 smithi174 ceph-mon[30085]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.6 KiB/s wr, 112 op/s 2024-03-11T22:19:52.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:51 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/1225213171' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T22:19:52.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:51 smithi183 ceph-mon[30906]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.6 KiB/s wr, 112 op/s 2024-03-11T22:19:52.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:51 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/1225213171' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T22:19:52.247 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:53.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:53 smithi174 ceph-mon[30085]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 39 KiB/s rd, 3.0 KiB/s wr, 93 op/s 2024-03-11T22:19:54.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:53 smithi027 ceph-mon[25797]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 39 KiB/s rd, 3.0 KiB/s wr, 93 op/s 2024-03-11T22:19:54.073 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:19:54.074 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":31,"fsid":"c71f9f14-dff4-11ee-95c8-87774f69a715","created":"2024-03-11T22:15:57.827890+0000","modified":"2024-03-11T22:19:40.410462+0000","last_up_change":"2024-03-11T22:19:19.144835+0000","last_in_change":"2024-03-11T22:19:00.736249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:19:20.934298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T22:19:31.765950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-11T22:19:33.725960+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-11T22:19:35.845013+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-11T22:19:37.902866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"816905d5-722e-43eb-9a39-0af0583b6c81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":328701664},{"type":"v1","addr":"172.21.15.27:6803","nonce":328701664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":328701664},{"type":"v1","addr":"172.21.15.27:6805","nonce":328701664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":328701664},{"type":"v1","addr":"172.21.15.27:6809","nonce":328701664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":328701664},{"type":"v1","addr":"172.21.15.27:6807","nonce":328701664}]},"public_addr":"172.21.15.27:6803/328701664","cluster_addr":"172.21.15.27:6805/328701664","heartbeat_back_addr":"172.21.15.27:6809/328701664","heartbeat_front_addr":"172.21.15.27:6807/328701664","state":["exists","up"]},{"osd":1,"uuid":"d1f90f41-442c-4b79-8c7b-6db7e8d41609","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":373410677},{"type":"v1","addr":"172.21.15.174:6801","nonce":373410677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":373410677},{"type":"v1","addr":"172.21.15.174:6803","nonce":373410677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":373410677},{"type":"v1","addr":"172.21.15.174:6807","nonce":373410677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":373410677},{"type":"v1","addr":"172.21.15.174:6805","nonce":373410677}]},"public_addr":"172.21.15.174:6801/373410677","cluster_addr":"172.21.15.174:6803/373410677","heartbeat_back_addr":"172.21.15.174:6807/373410677","heartbeat_front_addr":"172.21.15.174:6805/373410677","state":["exists","up"]},{"osd":2,"uuid":"1fbdab51-4cef-460e-9286-b049c5a63c23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":4216968028},{"type":"v1","addr":"172.21.15.183:6801","nonce":4216968028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":4216968028},{"type":"v1","addr":"172.21.15.183:6803","nonce":4216968028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":4216968028},{"type":"v1","addr":"172.21.15.183:6807","nonce":4216968028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":4216968028},{"type":"v1","addr":"172.21.15.183:6805","nonce":4216968028}]},"public_addr":"172.21.15.183:6801/4216968028","cluster_addr":"172.21.15.183:6803/4216968028","heartbeat_back_addr":"172.21.15.183:6807/4216968028","heartbeat_front_addr":"172.21.15.183:6805/4216968028","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:18:29.173381+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:18:53.145160+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:19:17.272070+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/990800086":"2024-03-12T22:16:54.388610+0000","172.21.15.27:6801/827022794":"2024-03-12T22:16:54.388610+0000","172.21.15.27:0/2902087996":"2024-03-12T22:16:54.388610+0000","172.21.15.27:0/2389474342":"2024-03-12T22:16:26.285486+0000","172.21.15.27:6800/827022794":"2024-03-12T22:16:54.388610+0000","172.21.15.27:0/1407031812":"2024-03-12T22:16:26.285486+0000","172.21.15.27:6800/1706259253":"2024-03-12T22:16:26.285486+0000","172.21.15.27:6801/1706259253":"2024-03-12T22:16:26.285486+0000","172.21.15.27:0/814977505":"2024-03-12T22:16:26.285486+0000","172.21.15.27:0/2606167595":"2024-03-12T22:16:54.388610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:19:54.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:53 smithi183 ceph-mon[30906]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 39 KiB/s rd, 3.0 KiB/s wr, 93 op/s 2024-03-11T22:19:54.761 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T22:19:54.761 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T22:19:55.004 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:54 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/1183806282' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:19:55.007 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:55.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:54 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/1183806282' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:19:55.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:54 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/1183806282' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:19:55.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:55 smithi174 ceph-mon[30085]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 11 KiB/s rd, 1023 B/s wr, 26 op/s 2024-03-11T22:19:56.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:55 smithi027 ceph-mon[25797]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 11 KiB/s rd, 1023 B/s wr, 26 op/s 2024-03-11T22:19:56.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:55 smithi183 ceph-mon[30906]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 11 KiB/s rd, 1023 B/s wr, 26 op/s 2024-03-11T22:19:56.153 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 podman[40472]: 2024-03-11 22:19:56.127626926 +0000 UTC m=+0.352095303 container start 6253f8395a1769acc6e588f89d8c7e1e68bb7dc449a967dbae0be1d914e15c85 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:19:56.153 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:19:56.151Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-11T22:19:56.153 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:19:56.151Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-11T22:19:56.153 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:19:56.151Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi174 (none))" 2024-03-11T22:19:56.153 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:19:56.151Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-11T22:19:56.153 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:19:56.151Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-11T22:19:56.505 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:19:56.153Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-11T22:19:56.505 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:19:56.153Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-11T22:19:56.506 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:19:56.154Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-11T22:19:56.506 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:19:56.154Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-11T22:19:56.506 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:19:56.157Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-11T22:19:56.506 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:19:56.157Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.344µs 2024-03-11T22:19:56.506 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:19:56.157Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-11T22:19:56.506 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:19:56.157Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-11T22:19:56.506 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:19:56.157Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=25.922µs wal_replay_duration=220.55µs wbl_replay_duration=135ns total_replay_duration=272.905µs 2024-03-11T22:19:56.506 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:19:56.158Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-11T22:19:56.506 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:19:56.158Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-11T22:19:56.506 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:19:56.158Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-11T22:19:56.506 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:19:56.176Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.668045ms db_storage=648ns remote_storage=1.03µs web_handler=471ns query_engine=459ns scrape=378.16µs scrape_sd=59.631µs notify=1.195µs notify_sd=1.271µs rules=17.049257ms tracing=5.381µs 2024-03-11T22:19:56.506 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:19:56.176Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-11T22:19:56.506 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:19:56.176Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-11T22:19:56.507 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 bash[40472]: 6253f8395a1769acc6e588f89d8c7e1e68bb7dc449a967dbae0be1d914e15c85 2024-03-11T22:19:56.507 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:19:56 smithi174 systemd[1]: Started Ceph prometheus.a for c71f9f14-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:19:56.932 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:19:56.933 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":31,"fsid":"c71f9f14-dff4-11ee-95c8-87774f69a715","created":"2024-03-11T22:15:57.827890+0000","modified":"2024-03-11T22:19:40.410462+0000","last_up_change":"2024-03-11T22:19:19.144835+0000","last_in_change":"2024-03-11T22:19:00.736249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:19:20.934298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T22:19:31.765950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-11T22:19:33.725960+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-11T22:19:35.845013+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-11T22:19:37.902866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"816905d5-722e-43eb-9a39-0af0583b6c81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":328701664},{"type":"v1","addr":"172.21.15.27:6803","nonce":328701664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":328701664},{"type":"v1","addr":"172.21.15.27:6805","nonce":328701664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":328701664},{"type":"v1","addr":"172.21.15.27:6809","nonce":328701664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":328701664},{"type":"v1","addr":"172.21.15.27:6807","nonce":328701664}]},"public_addr":"172.21.15.27:6803/328701664","cluster_addr":"172.21.15.27:6805/328701664","heartbeat_back_addr":"172.21.15.27:6809/328701664","heartbeat_front_addr":"172.21.15.27:6807/328701664","state":["exists","up"]},{"osd":1,"uuid":"d1f90f41-442c-4b79-8c7b-6db7e8d41609","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":373410677},{"type":"v1","addr":"172.21.15.174:6801","nonce":373410677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":373410677},{"type":"v1","addr":"172.21.15.174:6803","nonce":373410677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":373410677},{"type":"v1","addr":"172.21.15.174:6807","nonce":373410677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":373410677},{"type":"v1","addr":"172.21.15.174:6805","nonce":373410677}]},"public_addr":"172.21.15.174:6801/373410677","cluster_addr":"172.21.15.174:6803/373410677","heartbeat_back_addr":"172.21.15.174:6807/373410677","heartbeat_front_addr":"172.21.15.174:6805/373410677","state":["exists","up"]},{"osd":2,"uuid":"1fbdab51-4cef-460e-9286-b049c5a63c23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":4216968028},{"type":"v1","addr":"172.21.15.183:6801","nonce":4216968028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":4216968028},{"type":"v1","addr":"172.21.15.183:6803","nonce":4216968028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":4216968028},{"type":"v1","addr":"172.21.15.183:6807","nonce":4216968028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":4216968028},{"type":"v1","addr":"172.21.15.183:6805","nonce":4216968028}]},"public_addr":"172.21.15.183:6801/4216968028","cluster_addr":"172.21.15.183:6803/4216968028","heartbeat_back_addr":"172.21.15.183:6807/4216968028","heartbeat_front_addr":"172.21.15.183:6805/4216968028","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:18:29.173381+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:18:53.145160+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:19:17.272070+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/990800086":"2024-03-12T22:16:54.388610+0000","172.21.15.27:6801/827022794":"2024-03-12T22:16:54.388610+0000","172.21.15.27:0/2902087996":"2024-03-12T22:16:54.388610+0000","172.21.15.27:0/2389474342":"2024-03-12T22:16:26.285486+0000","172.21.15.27:6800/827022794":"2024-03-12T22:16:54.388610+0000","172.21.15.27:0/1407031812":"2024-03-12T22:16:26.285486+0000","172.21.15.27:6800/1706259253":"2024-03-12T22:16:26.285486+0000","172.21.15.27:6801/1706259253":"2024-03-12T22:16:26.285486+0000","172.21.15.27:0/814977505":"2024-03-12T22:16:26.285486+0000","172.21.15.27:0/2606167595":"2024-03-12T22:16:54.388610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:19:57.541 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:19:57 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: ignoring --setuser ceph since I am not root 2024-03-11T22:19:57.541 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:19:57 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: ignoring --setgroup ceph since I am not root 2024-03-11T22:19:57.541 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:57.542 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:57.542 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:57.542 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:57 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T22:19:57.542 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:57 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/479972146' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:19:57.543 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T22:19:57.544 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T22:19:57.544 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T22:19:57.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:57.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:57.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:57.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:57 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T22:19:57.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:57 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/479972146' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:19:57.604 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:19:57 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: ignoring --setuser ceph since I am not root 2024-03-11T22:19:57.604 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:19:57 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: ignoring --setgroup ceph since I am not root 2024-03-11T22:19:57.605 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:19:57 smithi183 ceph-mgr[33003]: -- 172.21.15.183:0/3762450254 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560c1b2b41a0 con 0x560c1b28b000 2024-03-11T22:19:57.754 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:57.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:57.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' 2024-03-11T22:19:57.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:57 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T22:19:57.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:57 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/479972146' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:19:57.755 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:19:57 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: ignoring --setuser ceph since I am not root 2024-03-11T22:19:57.755 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:19:57 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: ignoring --setgroup ceph since I am not root 2024-03-11T22:19:57.755 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:19:57 smithi174 ceph-mgr[31559]: -- 172.21.15.174:0/859582538 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55acaf0e41a0 con 0x55acaf0bac00 2024-03-11T22:19:57.830 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:57.830 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:58.242 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:19:57 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:19:57.990+0000 7ffa6fa92200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:19:58.242 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:19:58 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:19:58.090+0000 7ffa6fa92200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:19:58.255 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:19:57 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:19:57.892+0000 7fbf10b70200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:19:58.255 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:19:57 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:19:57.985+0000 7fbf10b70200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:19:58.289 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:19:57 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:19:57.895+0000 7f66b7282200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:19:58.289 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:19:57 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:19:57.987+0000 7f66b7282200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:19:58.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:58 smithi027 ceph-mon[25797]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 11 KiB/s rd, 1023 B/s wr, 26 op/s 2024-03-11T22:19:58.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:58 smithi027 ceph-mon[25797]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T22:19:58.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:19:58 smithi027 ceph-mon[25797]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-11T22:19:58.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:58 smithi183 ceph-mon[30906]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 11 KiB/s rd, 1023 B/s wr, 26 op/s 2024-03-11T22:19:58.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:58 smithi183 ceph-mon[30906]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T22:19:58.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:19:58 smithi183 ceph-mon[30906]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-11T22:19:58.618 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:58 smithi174 ceph-mon[30085]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 11 KiB/s rd, 1023 B/s wr, 26 op/s 2024-03-11T22:19:58.618 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:58 smithi174 ceph-mon[30085]: from='mgr.14152 172.21.15.27:0/2733516569' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T22:19:58.618 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:19:58 smithi174 ceph-mon[30085]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-11T22:19:58.948 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:19:58 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:19:58.616+0000 7fbf10b70200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:19:58.948 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:19:58 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:19:58.710+0000 7fbf10b70200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:19:58.948 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:19:58 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:19:58.851+0000 7fbf10b70200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:19:58.948 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:19:58 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:19:58.946+0000 7fbf10b70200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:19:58.950 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:19:58 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:19:58.620+0000 7f66b7282200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:19:58.950 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:19:58 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:19:58.712+0000 7f66b7282200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:19:58.950 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:19:58 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:19:58.852+0000 7f66b7282200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:19:58.950 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:19:58 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:19:58.948+0000 7f66b7282200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:19:59.114 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:19:58 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:19:58.785+0000 7ffa6fa92200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:19:59.114 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:19:58 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:19:58.882+0000 7ffa6fa92200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:19:59.114 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:19:59 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:19:59.030+0000 7ffa6fa92200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:19:59.254 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:19:59 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:19:59.027+0000 7fbf10b70200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:19:59.294 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:19:59 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:19:59.031+0000 7f66b7282200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:19:59.294 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:19:59 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:19:59.292+0000 7f66b7282200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:19:59.491 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:19:59 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:19:59.131+0000 7ffa6fa92200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:19:59.491 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:19:59 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:19:59.219+0000 7ffa6fa92200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:19:59.578 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:19:59 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:19:59.289+0000 7fbf10b70200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:19:59.578 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:19:59 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:19:59.397+0000 7fbf10b70200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:19:59.578 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:19:59 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:19:59.482+0000 7fbf10b70200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:19:59.578 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:19:59 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:19:59.576+0000 7fbf10b70200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:19:59.584 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:19:59 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:19:59.402+0000 7f66b7282200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:19:59.584 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:19:59 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:19:59.488+0000 7f66b7282200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:19:59.584 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:19:59 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:19:59.582+0000 7f66b7282200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:19:59.772 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:19:59 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:19:59.494+0000 7ffa6fa92200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:19:59.772 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:19:59 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:19:59.608+0000 7ffa6fa92200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:19:59.772 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:19:59 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:19:59.698+0000 7ffa6fa92200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:19:59.854 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:19:59 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:19:59.759+0000 7f66b7282200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:20:00.004 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:19:59 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:19:59.751+0000 7fbf10b70200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:20:00.078 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:00 smithi027 ceph-mon[25797]: overall HEALTH_OK 2024-03-11T22:20:00.078 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:19:59 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:19:59.801+0000 7ffa6fa92200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:20:00.078 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:19:59 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:19:59.985+0000 7ffa6fa92200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:20:00.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:00 smithi183 ceph-mon[30906]: overall HEALTH_OK 2024-03-11T22:20:00.504 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:00 smithi174 ceph-mon[30085]: overall HEALTH_OK 2024-03-11T22:20:01.005 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:00 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:20:00.705+0000 7fbf10b70200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:20:01.005 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:00 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:20:00.834+0000 7fbf10b70200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:20:01.103 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:00 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:20:00.710+0000 7f66b7282200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:20:01.104 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:00 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:20:00.838+0000 7f66b7282200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:20:01.145 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:01 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:20:01.033+0000 7ffa6fa92200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:20:01.408 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:01 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:20:01.176+0000 7ffa6fa92200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:20:01.456 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:01 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:20:01.168+0000 7fbf10b70200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:20:01.456 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:01 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:20:01.258+0000 7fbf10b70200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:20:01.456 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:01 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:20:01.454+0000 7fbf10b70200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:20:01.457 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:01 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:20:01.170+0000 7f66b7282200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:20:01.457 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:01 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:20:01.260+0000 7f66b7282200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:20:01.457 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:01 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:20:01.455+0000 7f66b7282200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:20:01.690 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:01 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:20:01.535+0000 7ffa6fa92200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:20:01.690 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:01 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:20:01.640+0000 7ffa6fa92200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:20:01.754 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:01 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:20:01.637+0000 7fbf10b70200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:20:01.854 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:01 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:20:01.638+0000 7f66b7282200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:20:02.022 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:01 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:20:01.850+0000 7ffa6fa92200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:20:02.182 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:01 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:20:01.919+0000 7f66b7282200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:20:02.182 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:02 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:20:02.180+0000 7f66b7282200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:20:02.208 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:01 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:20:01.919+0000 7fbf10b70200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:20:02.352 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:02 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:20:02.048+0000 7ffa6fa92200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:20:02.352 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:02 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:20:02.350+0000 7ffa6fa92200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:20:02.391 INFO:teuthology.orchestra.run.smithi027.stdout:77309411339 2024-03-11T22:20:02.391 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T22:20:02.504 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:02 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:20:02.207+0000 7fbf10b70200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:20:02.504 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:02 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:20:02.443+0000 7fbf10b70200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:20:02.603 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:02 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:20:02.414+0000 7f66b7282200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:20:02.897 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:02 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:20:02.639+0000 7ffa6fa92200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:20:02.898 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:02 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:20:02.896+0000 7ffa6fa92200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:20:03.185 INFO:teuthology.orchestra.run.smithi027.stdout:55834574863 2024-03-11T22:20:03.185 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T22:20:03.504 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:03 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:20:03.236+0000 7fbf10b70200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:20:03.603 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:03 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:20:03.203+0000 7f66b7282200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:20:03.944 INFO:teuthology.orchestra.run.smithi027.stdout:34359738389 2024-03-11T22:20:03.945 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T22:20:04.008 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:04.031 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:04.085 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:03 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:20:03.772+0000 7f66b7282200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:20:04.085 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:03 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:20:03.869+0000 7f66b7282200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:20:04.086 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:04 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:20:04.083+0000 7f66b7282200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:20:04.127 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:03 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:20:03.812+0000 7fbf10b70200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:20:04.127 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:03 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:20:03.910+0000 7fbf10b70200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:20:04.139 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:03 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:20:03.785+0000 7ffa6fa92200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:20:04.354 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:04 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: [11/Mar/2024:22:20:04] ENGINE Bus STARTING 2024-03-11T22:20:04.354 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:04 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: CherryPy Checker: 2024-03-11T22:20:04.354 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:04 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: The Application mounted at '' has an empty config. 2024-03-11T22:20:04.354 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:04 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: 2024-03-11T22:20:04.354 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:04 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: [11/Mar/2024:22:20:04] ENGINE Serving on http://:::9283 2024-03-11T22:20:04.354 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:04 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z[32980]: [11/Mar/2024:22:20:04] ENGINE Bus STARTED 2024-03-11T22:20:04.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:04 smithi183 ceph-mon[30906]: Standby manager daemon z restarted 2024-03-11T22:20:04.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:04 smithi183 ceph-mon[30906]: Standby manager daemon z started 2024-03-11T22:20:04.355 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:04 smithi183 ceph-mon[30906]: from='mgr.? 172.21.15.183:0/2324382125' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T22:20:04.355 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:04 smithi183 ceph-mon[30906]: from='mgr.? 172.21.15.183:0/2324382125' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:20:04.355 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:04 smithi183 ceph-mon[30906]: from='mgr.? 172.21.15.183:0/2324382125' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T22:20:04.355 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:04 smithi183 ceph-mon[30906]: from='mgr.? 172.21.15.183:0/2324382125' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:20:04.355 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:04 smithi183 ceph-mon[30906]: Standby manager daemon x restarted 2024-03-11T22:20:04.355 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:04 smithi183 ceph-mon[30906]: Standby manager daemon x started 2024-03-11T22:20:04.355 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:04 smithi183 ceph-mon[30906]: from='mgr.? 172.21.15.174:0/3915096922' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:20:04.355 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:04 smithi183 ceph-mon[30906]: from='mgr.? 172.21.15.174:0/3915096922' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:20:04.355 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:04 smithi183 ceph-mon[30906]: from='mgr.? 172.21.15.174:0/3915096922' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:20:04.355 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:04 smithi183 ceph-mon[30906]: from='mgr.? 172.21.15.174:0/3915096922' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:20:04.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:04 smithi027 ceph-mon[25797]: Standby manager daemon z restarted 2024-03-11T22:20:04.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:04 smithi027 ceph-mon[25797]: Standby manager daemon z started 2024-03-11T22:20:04.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:04 smithi027 ceph-mon[25797]: from='mgr.? 172.21.15.183:0/2324382125' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T22:20:04.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:04 smithi027 ceph-mon[25797]: from='mgr.? 172.21.15.183:0/2324382125' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:20:04.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:04 smithi027 ceph-mon[25797]: from='mgr.? 172.21.15.183:0/2324382125' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T22:20:04.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:04 smithi027 ceph-mon[25797]: from='mgr.? 172.21.15.183:0/2324382125' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:20:04.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:04 smithi027 ceph-mon[25797]: Standby manager daemon x restarted 2024-03-11T22:20:04.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:04 smithi027 ceph-mon[25797]: Standby manager daemon x started 2024-03-11T22:20:04.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:04 smithi027 ceph-mon[25797]: from='mgr.? 172.21.15.174:0/3915096922' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:20:04.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:04 smithi027 ceph-mon[25797]: from='mgr.? 172.21.15.174:0/3915096922' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:20:04.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:04 smithi027 ceph-mon[25797]: from='mgr.? 172.21.15.174:0/3915096922' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:20:04.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:04 smithi027 ceph-mon[25797]: from='mgr.? 172.21.15.174:0/3915096922' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:20:04.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:04 smithi174 ceph-mon[30085]: Standby manager daemon z restarted 2024-03-11T22:20:04.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:04 smithi174 ceph-mon[30085]: Standby manager daemon z started 2024-03-11T22:20:04.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:04 smithi174 ceph-mon[30085]: from='mgr.? 172.21.15.183:0/2324382125' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T22:20:04.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:04 smithi174 ceph-mon[30085]: from='mgr.? 172.21.15.183:0/2324382125' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:20:04.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:04 smithi174 ceph-mon[30085]: from='mgr.? 172.21.15.183:0/2324382125' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T22:20:04.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:04 smithi174 ceph-mon[30085]: from='mgr.? 172.21.15.183:0/2324382125' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:20:04.506 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:04 smithi174 ceph-mon[30085]: Standby manager daemon x restarted 2024-03-11T22:20:04.506 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:04 smithi174 ceph-mon[30085]: Standby manager daemon x started 2024-03-11T22:20:04.506 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:04 smithi174 ceph-mon[30085]: from='mgr.? 172.21.15.174:0/3915096922' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:20:04.506 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:04 smithi174 ceph-mon[30085]: from='mgr.? 172.21.15.174:0/3915096922' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:20:04.506 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:04 smithi174 ceph-mon[30085]: from='mgr.? 172.21.15.174:0/3915096922' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:20:04.506 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:04 smithi174 ceph-mon[30085]: from='mgr.? 172.21.15.174:0/3915096922' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:20:04.506 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:04 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:20:04.125+0000 7fbf10b70200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:20:04.506 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:04 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: [11/Mar/2024:22:20:04] ENGINE Bus STARTING 2024-03-11T22:20:04.506 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:04 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: CherryPy Checker: 2024-03-11T22:20:04.506 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:04 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: The Application mounted at '' has an empty config. 2024-03-11T22:20:04.506 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:04 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: 2024-03-11T22:20:04.506 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:04 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: [11/Mar/2024:22:20:04] ENGINE Serving on http://:::9283 2024-03-11T22:20:04.507 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:04 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x[31536]: [11/Mar/2024:22:20:04] ENGINE Bus STARTED 2024-03-11T22:20:04.658 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:04 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:20:04.433+0000 7ffa6fa92200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:20:04.659 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:04 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:20:04.548+0000 7ffa6fa92200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:20:04.960 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:04 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:20:04.784+0000 7ffa6fa92200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:20:04.960 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:04 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: [11/Mar/2024:22:20:04] ENGINE Bus STARTING 2024-03-11T22:20:05.238 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-11T22:20:05.238 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: Active manager daemon y restarted 2024-03-11T22:20:05.238 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: Activating manager daemon y 2024-03-11T22:20:05.238 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: osdmap e32: 3 total, 3 up, 3 in 2024-03-11T22:20:05.238 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: mgrmap e19: y(active, starting, since 0.0100138s), standbys: x, z 2024-03-11T22:20:05.239 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:20:05.239 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:20:05.239 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:20:05.239 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:20:05.239 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:20:05.239 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T22:20:05.239 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:20:05.239 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:20:05.239 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:20:05.239 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:20:05.239 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:20:05.239 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:20:05.239 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: Manager daemon y is now available 2024-03-11T22:20:05.239 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:20:05.239 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:20:05.240 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:05.240 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:20:05.240 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:20:05.240 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:05 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:20:05.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-11T22:20:05.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: Active manager daemon y restarted 2024-03-11T22:20:05.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: Activating manager daemon y 2024-03-11T22:20:05.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: osdmap e32: 3 total, 3 up, 3 in 2024-03-11T22:20:05.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: mgrmap e19: y(active, starting, since 0.0100138s), standbys: x, z 2024-03-11T22:20:05.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:20:05.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:20:05.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:20:05.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:20:05.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:20:05.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T22:20:05.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:20:05.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:20:05.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:20:05.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:20:05.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:20:05.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:20:05.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: Manager daemon y is now available 2024-03-11T22:20:05.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:20:05.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:20:05.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:05.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:20:05.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:20:05.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:05 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:20:05.272 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: CherryPy Checker: 2024-03-11T22:20:05.272 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: The Application mounted at '' has an empty config. 2024-03-11T22:20:05.272 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:20:05.272 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: [11/Mar/2024:22:20:05] ENGINE Serving on http://:::9283 2024-03-11T22:20:05.272 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: [11/Mar/2024:22:20:05] ENGINE Bus STARTED 2024-03-11T22:20:05.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-11T22:20:05.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: Active manager daemon y restarted 2024-03-11T22:20:05.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: Activating manager daemon y 2024-03-11T22:20:05.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: osdmap e32: 3 total, 3 up, 3 in 2024-03-11T22:20:05.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: mgrmap e19: y(active, starting, since 0.0100138s), standbys: x, z 2024-03-11T22:20:05.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:20:05.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:20:05.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:20:05.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:20:05.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:20:05.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T22:20:05.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:20:05.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:20:05.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:20:05.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:20:05.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:20:05.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:20:05.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: Manager daemon y is now available 2024-03-11T22:20:05.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:20:05.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:20:05.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:05.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:20:05.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:20:05.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:05 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:20:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:07.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:06 smithi183 ceph-mon[30906]: [11/Mar/2024:22:20:05] ENGINE Bus STARTING 2024-03-11T22:20:07.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:06 smithi183 ceph-mon[30906]: [11/Mar/2024:22:20:05] ENGINE Serving on http://172.21.15.27:8765 2024-03-11T22:20:07.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:06 smithi183 ceph-mon[30906]: [11/Mar/2024:22:20:05] ENGINE Serving on https://172.21.15.27:7150 2024-03-11T22:20:07.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:06 smithi183 ceph-mon[30906]: [11/Mar/2024:22:20:05] ENGINE Bus STARTED 2024-03-11T22:20:07.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:06 smithi183 ceph-mon[30906]: mgrmap e20: y(active, since 1.016s), standbys: x, z 2024-03-11T22:20:07.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:06 smithi183 ceph-mon[30906]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:07.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:06 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:07.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:06 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:07.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:06 smithi174 ceph-mon[30085]: [11/Mar/2024:22:20:05] ENGINE Bus STARTING 2024-03-11T22:20:07.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:06 smithi174 ceph-mon[30085]: [11/Mar/2024:22:20:05] ENGINE Serving on http://172.21.15.27:8765 2024-03-11T22:20:07.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:06 smithi174 ceph-mon[30085]: [11/Mar/2024:22:20:05] ENGINE Serving on https://172.21.15.27:7150 2024-03-11T22:20:07.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:06 smithi174 ceph-mon[30085]: [11/Mar/2024:22:20:05] ENGINE Bus STARTED 2024-03-11T22:20:07.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:06 smithi174 ceph-mon[30085]: mgrmap e20: y(active, since 1.016s), standbys: x, z 2024-03-11T22:20:07.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:06 smithi174 ceph-mon[30085]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:07.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:06 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:07.255 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:06 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:07.265 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:06 smithi027 ceph-mon[25797]: [11/Mar/2024:22:20:05] ENGINE Bus STARTING 2024-03-11T22:20:07.266 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:06 smithi027 ceph-mon[25797]: [11/Mar/2024:22:20:05] ENGINE Serving on http://172.21.15.27:8765 2024-03-11T22:20:07.266 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:06 smithi027 ceph-mon[25797]: [11/Mar/2024:22:20:05] ENGINE Serving on https://172.21.15.27:7150 2024-03-11T22:20:07.266 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:06 smithi027 ceph-mon[25797]: [11/Mar/2024:22:20:05] ENGINE Bus STARTED 2024-03-11T22:20:07.266 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:06 smithi027 ceph-mon[25797]: mgrmap e20: y(active, since 1.016s), standbys: x, z 2024-03-11T22:20:07.266 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:06 smithi027 ceph-mon[25797]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:07.266 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:06 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:07.266 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:06 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:07.809 INFO:teuthology.orchestra.run.smithi027.stdout:77309411340 2024-03-11T22:20:07.911 INFO:teuthology.orchestra.run.smithi027.stdout:55834574864 2024-03-11T22:20:08.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:08 smithi027 ceph-mon[25797]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:08.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:08 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:08.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:08 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:08.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:08 smithi027 ceph-mon[25797]: mgrmap e21: y(active, since 2s), standbys: x, z 2024-03-11T22:20:08.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:08 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:08.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:08 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:08.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:08 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/3115781717' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T22:20:08.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:08 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/1492034647' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T22:20:08.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:08 smithi183 ceph-mon[30906]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:08.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:08 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:08.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:08 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:08.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:08 smithi183 ceph-mon[30906]: mgrmap e21: y(active, since 2s), standbys: x, z 2024-03-11T22:20:08.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:08 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:08.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:08 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:08.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:08 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/3115781717' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T22:20:08.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:08 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/1492034647' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T22:20:08.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:08 smithi174 ceph-mon[30085]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:08.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:08 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:08.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:08 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:08.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:08 smithi174 ceph-mon[30085]: mgrmap e21: y(active, since 2s), standbys: x, z 2024-03-11T22:20:08.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:08 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:08.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:08 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:08.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:08 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/3115781717' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T22:20:08.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:08 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/1492034647' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T22:20:09.138 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574863 got 55834574864 for osd.1 2024-03-11T22:20:09.138 DEBUG:teuthology.parallel:result is None 2024-03-11T22:20:09.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:09 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:09.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:09 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:09.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:09 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:09.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:09 smithi174 ceph-mon[30085]: Adjusting osd_memory_target on smithi174 to 16064M 2024-03-11T22:20:09.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:09 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:09.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:09 smithi174 ceph-mon[30085]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:09.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:09 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:09.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:09 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:09.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:09 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:09.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:09 smithi027 ceph-mon[25797]: Adjusting osd_memory_target on smithi174 to 16064M 2024-03-11T22:20:09.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:09 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:09.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:09 smithi027 ceph-mon[25797]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:09.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:09 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:09.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:09 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:09.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:09 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:09.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:09 smithi183 ceph-mon[30906]: Adjusting osd_memory_target on smithi174 to 16064M 2024-03-11T22:20:09.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:09 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:09.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:09 smithi183 ceph-mon[30906]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:09.860 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411339 got 77309411340 for osd.2 2024-03-11T22:20:09.860 DEBUG:teuthology.parallel:result is None 2024-03-11T22:20:10.578 INFO:teuthology.orchestra.run.smithi027.stdout:34359738390 2024-03-11T22:20:10.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:10 smithi174 ceph-mon[30085]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-11T22:20:10.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:10 smithi027 ceph-mon[25797]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-11T22:20:10.772 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:10 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: ::ffff:172.21.15.174 - - [11/Mar/2024:22:20:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:20:10.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:10 smithi183 ceph-mon[30906]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-11T22:20:11.139 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738389 got 34359738390 for osd.0 2024-03-11T22:20:11.139 DEBUG:teuthology.parallel:result is None 2024-03-11T22:20:11.139 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T22:20:11.139 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:20:11.403 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:11.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:11 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:11.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:11 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:11.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:11 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/2368953335' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T22:20:11.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:11 smithi174 ceph-mon[30085]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:11.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:11 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:11.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:11 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:11.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:11 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/2368953335' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T22:20:11.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:11 smithi027 ceph-mon[25797]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:11.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:11 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:11.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:11 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:11.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:11 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/2368953335' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T22:20:11.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:11 smithi183 ceph-mon[30906]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:13.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:13 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:13.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:13 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:13.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:13 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:13.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:13 smithi027 ceph-mon[25797]: Adjusting osd_memory_target on smithi027 to 16064M 2024-03-11T22:20:13.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:13 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:13.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:13 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:13.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:13 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:20:13.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:13 smithi027 ceph-mon[25797]: Updating smithi027:/etc/ceph/ceph.conf 2024-03-11T22:20:13.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:13 smithi027 ceph-mon[25797]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-11T22:20:13.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:13 smithi027 ceph-mon[25797]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-11T22:20:13.481 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:20:13.483 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-11T22:20:13.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:13 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:13.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:13 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:13.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:13 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:13.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:13 smithi174 ceph-mon[30085]: Adjusting osd_memory_target on smithi027 to 16064M 2024-03-11T22:20:13.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:13 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:13.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:13 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:13.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:13 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:20:13.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:13 smithi174 ceph-mon[30085]: Updating smithi027:/etc/ceph/ceph.conf 2024-03-11T22:20:13.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:13 smithi174 ceph-mon[30085]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-11T22:20:13.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:13 smithi174 ceph-mon[30085]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-11T22:20:13.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:13 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:13.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:13 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:13.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:13 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:13.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:13 smithi183 ceph-mon[30906]: Adjusting osd_memory_target on smithi027 to 16064M 2024-03-11T22:20:13.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:13 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:13.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:13 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:13.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:13 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:20:13.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:13 smithi183 ceph-mon[30906]: Updating smithi027:/etc/ceph/ceph.conf 2024-03-11T22:20:13.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:13 smithi183 ceph-mon[30906]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-11T22:20:13.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:13 smithi183 ceph-mon[30906]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-11T22:20:14.072 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":7,"stamp":"2024-03-11T22:20:12.814848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595908,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96636,"kb_used_data":3108,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126532,"statfs":{"total":287972524032,"available":287873568768,"internally_reserved":0,"allocated":3182592,"data_stored":2109498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":95743312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.997247"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806424+0000","last_change":"2024-03-11T22:19:39.421116+0000","last_active":"2024-03-11T22:20:04.806424+0000","last_peered":"2024-03-11T22:20:04.806424+0000","last_clean":"2024-03-11T22:20:04.806424+0000","last_became_active":"2024-03-11T22:19:39.420843+0000","last_became_peered":"2024-03-11T22:19:39.420843+0000","last_unstale":"2024-03-11T22:20:04.806424+0000","last_undegraded":"2024-03-11T22:20:04.806424+0000","last_fullsized":"2024-03-11T22:20:04.806424+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:53:32.203791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805175+0000","last_change":"2024-03-11T22:19:37.406058+0000","last_active":"2024-03-11T22:20:04.805175+0000","last_peered":"2024-03-11T22:20:04.805175+0000","last_clean":"2024-03-11T22:20:04.805175+0000","last_became_active":"2024-03-11T22:19:37.405848+0000","last_became_peered":"2024-03-11T22:19:37.405848+0000","last_unstale":"2024-03-11T22:20:04.805175+0000","last_undegraded":"2024-03-11T22:20:04.805175+0000","last_fullsized":"2024-03-11T22:20:04.805175+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:58:10.467468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806785+0000","last_change":"2024-03-11T22:19:33.222933+0000","last_active":"2024-03-11T22:20:04.806785+0000","last_peered":"2024-03-11T22:20:04.806785+0000","last_clean":"2024-03-11T22:20:04.806785+0000","last_became_active":"2024-03-11T22:19:33.222624+0000","last_became_peered":"2024-03-11T22:19:33.222624+0000","last_unstale":"2024-03-11T22:20:04.806785+0000","last_undegraded":"2024-03-11T22:20:04.806785+0000","last_fullsized":"2024-03-11T22:20:04.806785+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:25:02.265071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806100+0000","last_change":"2024-03-11T22:19:35.407660+0000","last_active":"2024-03-11T22:20:04.806100+0000","last_peered":"2024-03-11T22:20:04.806100+0000","last_clean":"2024-03-11T22:20:04.806100+0000","last_became_active":"2024-03-11T22:19:35.407385+0000","last_became_peered":"2024-03-11T22:19:35.407385+0000","last_unstale":"2024-03-11T22:20:04.806100+0000","last_undegraded":"2024-03-11T22:20:04.806100+0000","last_fullsized":"2024-03-11T22:20:04.806100+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:59:48.788844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804004+0000","last_change":"2024-03-11T22:19:39.421158+0000","last_active":"2024-03-11T22:20:04.804004+0000","last_peered":"2024-03-11T22:20:04.804004+0000","last_clean":"2024-03-11T22:20:04.804004+0000","last_became_active":"2024-03-11T22:19:39.420791+0000","last_became_peered":"2024-03-11T22:19:39.420791+0000","last_unstale":"2024-03-11T22:20:04.804004+0000","last_undegraded":"2024-03-11T22:20:04.804004+0000","last_fullsized":"2024-03-11T22:20:04.804004+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:07:24.235523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805884+0000","last_change":"2024-03-11T22:19:37.411798+0000","last_active":"2024-03-11T22:20:04.805884+0000","last_peered":"2024-03-11T22:20:04.805884+0000","last_clean":"2024-03-11T22:20:04.805884+0000","last_became_active":"2024-03-11T22:19:37.410719+0000","last_became_peered":"2024-03-11T22:19:37.410719+0000","last_unstale":"2024-03-11T22:20:04.805884+0000","last_undegraded":"2024-03-11T22:20:04.805884+0000","last_fullsized":"2024-03-11T22:20:04.805884+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:43:09.846210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806779+0000","last_change":"2024-03-11T22:19:33.223155+0000","last_active":"2024-03-11T22:20:04.806779+0000","last_peered":"2024-03-11T22:20:04.806779+0000","last_clean":"2024-03-11T22:20:04.806779+0000","last_became_active":"2024-03-11T22:19:33.222732+0000","last_became_peered":"2024-03-11T22:19:33.222732+0000","last_unstale":"2024-03-11T22:20:04.806779+0000","last_undegraded":"2024-03-11T22:20:04.806779+0000","last_fullsized":"2024-03-11T22:20:04.806779+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:26:56.930204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806718+0000","last_change":"2024-03-11T22:19:35.408249+0000","last_active":"2024-03-11T22:20:04.806718+0000","last_peered":"2024-03-11T22:20:04.806718+0000","last_clean":"2024-03-11T22:20:04.806718+0000","last_became_active":"2024-03-11T22:19:35.407907+0000","last_became_peered":"2024-03-11T22:19:35.407907+0000","last_unstale":"2024-03-11T22:20:04.806718+0000","last_undegraded":"2024-03-11T22:20:04.806718+0000","last_fullsized":"2024-03-11T22:20:04.806718+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:13:12.861495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805599+0000","last_change":"2024-03-11T22:19:39.423823+0000","last_active":"2024-03-11T22:20:04.805599+0000","last_peered":"2024-03-11T22:20:04.805599+0000","last_clean":"2024-03-11T22:20:04.805599+0000","last_became_active":"2024-03-11T22:19:39.423538+0000","last_became_peered":"2024-03-11T22:19:39.423538+0000","last_unstale":"2024-03-11T22:20:04.805599+0000","last_undegraded":"2024-03-11T22:20:04.805599+0000","last_fullsized":"2024-03-11T22:20:04.805599+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:28:46.921390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807717+0000","last_change":"2024-03-11T22:19:37.412592+0000","last_active":"2024-03-11T22:20:04.807717+0000","last_peered":"2024-03-11T22:20:04.807717+0000","last_clean":"2024-03-11T22:20:04.807717+0000","last_became_active":"2024-03-11T22:19:37.412130+0000","last_became_peered":"2024-03-11T22:19:37.412130+0000","last_unstale":"2024-03-11T22:20:04.807717+0000","last_undegraded":"2024-03-11T22:20:04.807717+0000","last_fullsized":"2024-03-11T22:20:04.807717+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:46:46.519282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808683+0000","last_change":"2024-03-11T22:19:33.224984+0000","last_active":"2024-03-11T22:20:04.808683+0000","last_peered":"2024-03-11T22:20:04.808683+0000","last_clean":"2024-03-11T22:20:04.808683+0000","last_became_active":"2024-03-11T22:19:33.224251+0000","last_became_peered":"2024-03-11T22:19:33.224251+0000","last_unstale":"2024-03-11T22:20:04.808683+0000","last_undegraded":"2024-03-11T22:20:04.808683+0000","last_fullsized":"2024-03-11T22:20:04.808683+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:04:13.898069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808397+0000","last_change":"2024-03-11T22:19:35.403297+0000","last_active":"2024-03-11T22:20:04.808397+0000","last_peered":"2024-03-11T22:20:04.808397+0000","last_clean":"2024-03-11T22:20:04.808397+0000","last_became_active":"2024-03-11T22:19:35.401404+0000","last_became_peered":"2024-03-11T22:19:35.401404+0000","last_unstale":"2024-03-11T22:20:04.808397+0000","last_undegraded":"2024-03-11T22:20:04.808397+0000","last_fullsized":"2024-03-11T22:20:04.808397+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:39.972534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805678+0000","last_change":"2024-03-11T22:19:39.421824+0000","last_active":"2024-03-11T22:20:04.805678+0000","last_peered":"2024-03-11T22:20:04.805678+0000","last_clean":"2024-03-11T22:20:04.805678+0000","last_became_active":"2024-03-11T22:19:39.421402+0000","last_became_peered":"2024-03-11T22:19:39.421402+0000","last_unstale":"2024-03-11T22:20:04.805678+0000","last_undegraded":"2024-03-11T22:20:04.805678+0000","last_fullsized":"2024-03-11T22:20:04.805678+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:01:58.087718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805810+0000","last_change":"2024-03-11T22:19:37.412169+0000","last_active":"2024-03-11T22:20:04.805810+0000","last_peered":"2024-03-11T22:20:04.805810+0000","last_clean":"2024-03-11T22:20:04.805810+0000","last_became_active":"2024-03-11T22:19:37.410836+0000","last_became_peered":"2024-03-11T22:19:37.410836+0000","last_unstale":"2024-03-11T22:20:04.805810+0000","last_undegraded":"2024-03-11T22:20:04.805810+0000","last_fullsized":"2024-03-11T22:20:04.805810+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:30:19.937932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807430+0000","last_change":"2024-03-11T22:19:33.222677+0000","last_active":"2024-03-11T22:20:04.807430+0000","last_peered":"2024-03-11T22:20:04.807430+0000","last_clean":"2024-03-11T22:20:04.807430+0000","last_became_active":"2024-03-11T22:19:33.221752+0000","last_became_peered":"2024-03-11T22:19:33.221752+0000","last_unstale":"2024-03-11T22:20:04.807430+0000","last_undegraded":"2024-03-11T22:20:04.807430+0000","last_fullsized":"2024-03-11T22:20:04.807430+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:11:40.222643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"31'12","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808499+0000","last_change":"2024-03-11T22:19:35.398123+0000","last_active":"2024-03-11T22:20:04.808499+0000","last_peered":"2024-03-11T22:20:04.808499+0000","last_clean":"2024-03-11T22:20:04.808499+0000","last_became_active":"2024-03-11T22:19:35.397830+0000","last_became_peered":"2024-03-11T22:19:35.397830+0000","last_unstale":"2024-03-11T22:20:04.808499+0000","last_undegraded":"2024-03-11T22:20:04.808499+0000","last_fullsized":"2024-03-11T22:20:04.808499+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:56:29.329916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807454+0000","last_change":"2024-03-11T22:19:39.423075+0000","last_active":"2024-03-11T22:20:04.807454+0000","last_peered":"2024-03-11T22:20:04.807454+0000","last_clean":"2024-03-11T22:20:04.807454+0000","last_became_active":"2024-03-11T22:19:39.422424+0000","last_became_peered":"2024-03-11T22:19:39.422424+0000","last_unstale":"2024-03-11T22:20:04.807454+0000","last_undegraded":"2024-03-11T22:20:04.807454+0000","last_fullsized":"2024-03-11T22:20:04.807454+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:51:28.274829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807722+0000","last_change":"2024-03-11T22:19:37.407427+0000","last_active":"2024-03-11T22:20:04.807722+0000","last_peered":"2024-03-11T22:20:04.807722+0000","last_clean":"2024-03-11T22:20:04.807722+0000","last_became_active":"2024-03-11T22:19:37.407104+0000","last_became_peered":"2024-03-11T22:19:37.407104+0000","last_unstale":"2024-03-11T22:20:04.807722+0000","last_undegraded":"2024-03-11T22:20:04.807722+0000","last_fullsized":"2024-03-11T22:20:04.807722+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:34:01.512236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806860+0000","last_change":"2024-03-11T22:19:33.223374+0000","last_active":"2024-03-11T22:20:04.806860+0000","last_peered":"2024-03-11T22:20:04.806860+0000","last_clean":"2024-03-11T22:20:04.806860+0000","last_became_active":"2024-03-11T22:19:33.221665+0000","last_became_peered":"2024-03-11T22:19:33.221665+0000","last_unstale":"2024-03-11T22:20:04.806860+0000","last_undegraded":"2024-03-11T22:20:04.806860+0000","last_fullsized":"2024-03-11T22:20:04.806860+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:48:00.757046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806346+0000","last_change":"2024-03-11T22:19:35.397986+0000","last_active":"2024-03-11T22:20:04.806346+0000","last_peered":"2024-03-11T22:20:04.806346+0000","last_clean":"2024-03-11T22:20:04.806346+0000","last_became_active":"2024-03-11T22:19:35.397623+0000","last_became_peered":"2024-03-11T22:19:35.397623+0000","last_unstale":"2024-03-11T22:20:04.806346+0000","last_undegraded":"2024-03-11T22:20:04.806346+0000","last_fullsized":"2024-03-11T22:20:04.806346+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:35:29.027191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806045+0000","last_change":"2024-03-11T22:19:39.426841+0000","last_active":"2024-03-11T22:20:04.806045+0000","last_peered":"2024-03-11T22:20:04.806045+0000","last_clean":"2024-03-11T22:20:04.806045+0000","last_became_active":"2024-03-11T22:19:39.426369+0000","last_became_peered":"2024-03-11T22:19:39.426369+0000","last_unstale":"2024-03-11T22:20:04.806045+0000","last_undegraded":"2024-03-11T22:20:04.806045+0000","last_fullsized":"2024-03-11T22:20:04.806045+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:13:42.460874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807211+0000","last_change":"2024-03-11T22:19:37.407440+0000","last_active":"2024-03-11T22:20:04.807211+0000","last_peered":"2024-03-11T22:20:04.807211+0000","last_clean":"2024-03-11T22:20:04.807211+0000","last_became_active":"2024-03-11T22:19:37.407049+0000","last_became_peered":"2024-03-11T22:19:37.407049+0000","last_unstale":"2024-03-11T22:20:04.807211+0000","last_undegraded":"2024-03-11T22:20:04.807211+0000","last_fullsized":"2024-03-11T22:20:04.807211+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:22:26.161223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806883+0000","last_change":"2024-03-11T22:19:33.227456+0000","last_active":"2024-03-11T22:20:04.806883+0000","last_peered":"2024-03-11T22:20:04.806883+0000","last_clean":"2024-03-11T22:20:04.806883+0000","last_became_active":"2024-03-11T22:19:33.227147+0000","last_became_peered":"2024-03-11T22:19:33.227147+0000","last_unstale":"2024-03-11T22:20:04.806883+0000","last_undegraded":"2024-03-11T22:20:04.806883+0000","last_fullsized":"2024-03-11T22:20:04.806883+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:43:14.374168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'5","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806741+0000","last_change":"2024-03-11T22:19:35.397709+0000","last_active":"2024-03-11T22:20:04.806741+0000","last_peered":"2024-03-11T22:20:04.806741+0000","last_clean":"2024-03-11T22:20:04.806741+0000","last_became_active":"2024-03-11T22:19:35.397328+0000","last_became_peered":"2024-03-11T22:19:35.397328+0000","last_unstale":"2024-03-11T22:20:04.806741+0000","last_undegraded":"2024-03-11T22:20:04.806741+0000","last_fullsized":"2024-03-11T22:20:04.806741+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:06:21.456714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805494+0000","last_change":"2024-03-11T22:19:39.426413+0000","last_active":"2024-03-11T22:20:04.805494+0000","last_peered":"2024-03-11T22:20:04.805494+0000","last_clean":"2024-03-11T22:20:04.805494+0000","last_became_active":"2024-03-11T22:19:39.425543+0000","last_became_peered":"2024-03-11T22:19:39.425543+0000","last_unstale":"2024-03-11T22:20:04.805494+0000","last_undegraded":"2024-03-11T22:20:04.805494+0000","last_fullsized":"2024-03-11T22:20:04.805494+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:56:37.440502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"31'8","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808025+0000","last_change":"2024-03-11T22:19:37.406900+0000","last_active":"2024-03-11T22:20:04.808025+0000","last_peered":"2024-03-11T22:20:04.808025+0000","last_clean":"2024-03-11T22:20:04.808025+0000","last_became_active":"2024-03-11T22:19:37.406387+0000","last_became_peered":"2024-03-11T22:19:37.406387+0000","last_unstale":"2024-03-11T22:20:04.808025+0000","last_undegraded":"2024-03-11T22:20:04.808025+0000","last_fullsized":"2024-03-11T22:20:04.808025+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:44:42.280262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804220+0000","last_change":"2024-03-11T22:19:33.223048+0000","last_active":"2024-03-11T22:20:04.804220+0000","last_peered":"2024-03-11T22:20:04.804220+0000","last_clean":"2024-03-11T22:20:04.804220+0000","last_became_active":"2024-03-11T22:19:33.222760+0000","last_became_peered":"2024-03-11T22:19:33.222760+0000","last_unstale":"2024-03-11T22:20:04.804220+0000","last_undegraded":"2024-03-11T22:20:04.804220+0000","last_fullsized":"2024-03-11T22:20:04.804220+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:28:40.522004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.803679+0000","last_change":"2024-03-11T22:19:35.406928+0000","last_active":"2024-03-11T22:20:04.803679+0000","last_peered":"2024-03-11T22:20:04.803679+0000","last_clean":"2024-03-11T22:20:04.803679+0000","last_became_active":"2024-03-11T22:19:35.406155+0000","last_became_peered":"2024-03-11T22:19:35.406155+0000","last_unstale":"2024-03-11T22:20:04.803679+0000","last_undegraded":"2024-03-11T22:20:04.803679+0000","last_fullsized":"2024-03-11T22:20:04.803679+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:22:34.927007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806299+0000","last_change":"2024-03-11T22:19:39.423744+0000","last_active":"2024-03-11T22:20:04.806299+0000","last_peered":"2024-03-11T22:20:04.806299+0000","last_clean":"2024-03-11T22:20:04.806299+0000","last_became_active":"2024-03-11T22:19:39.423369+0000","last_became_peered":"2024-03-11T22:19:39.423369+0000","last_unstale":"2024-03-11T22:20:04.806299+0000","last_undegraded":"2024-03-11T22:20:04.806299+0000","last_fullsized":"2024-03-11T22:20:04.806299+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:24:11.705110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805799+0000","last_change":"2024-03-11T22:19:37.409605+0000","last_active":"2024-03-11T22:20:04.805799+0000","last_peered":"2024-03-11T22:20:04.805799+0000","last_clean":"2024-03-11T22:20:04.805799+0000","last_became_active":"2024-03-11T22:19:37.409209+0000","last_became_peered":"2024-03-11T22:19:37.409209+0000","last_unstale":"2024-03-11T22:20:04.805799+0000","last_undegraded":"2024-03-11T22:20:04.805799+0000","last_fullsized":"2024-03-11T22:20:04.805799+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:36:17.210201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805190+0000","last_change":"2024-03-11T22:19:33.224855+0000","last_active":"2024-03-11T22:20:04.805190+0000","last_peered":"2024-03-11T22:20:04.805190+0000","last_clean":"2024-03-11T22:20:04.805190+0000","last_became_active":"2024-03-11T22:19:33.224355+0000","last_became_peered":"2024-03-11T22:19:33.224355+0000","last_unstale":"2024-03-11T22:20:04.805190+0000","last_undegraded":"2024-03-11T22:20:04.805190+0000","last_fullsized":"2024-03-11T22:20:04.805190+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:01:30.076824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.803832+0000","last_change":"2024-03-11T22:19:35.406509+0000","last_active":"2024-03-11T22:20:04.803832+0000","last_peered":"2024-03-11T22:20:04.803832+0000","last_clean":"2024-03-11T22:20:04.803832+0000","last_became_active":"2024-03-11T22:19:35.406114+0000","last_became_peered":"2024-03-11T22:19:35.406114+0000","last_unstale":"2024-03-11T22:20:04.803832+0000","last_undegraded":"2024-03-11T22:20:04.803832+0000","last_fullsized":"2024-03-11T22:20:04.803832+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:17:32.892908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806581+0000","last_change":"2024-03-11T22:19:39.424762+0000","last_active":"2024-03-11T22:20:04.806581+0000","last_peered":"2024-03-11T22:20:04.806581+0000","last_clean":"2024-03-11T22:20:04.806581+0000","last_became_active":"2024-03-11T22:19:39.424493+0000","last_became_peered":"2024-03-11T22:19:39.424493+0000","last_unstale":"2024-03-11T22:20:04.806581+0000","last_undegraded":"2024-03-11T22:20:04.806581+0000","last_fullsized":"2024-03-11T22:20:04.806581+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:42:01.588578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.803825+0000","last_change":"2024-03-11T22:19:37.409872+0000","last_active":"2024-03-11T22:20:04.803825+0000","last_peered":"2024-03-11T22:20:04.803825+0000","last_clean":"2024-03-11T22:20:04.803825+0000","last_became_active":"2024-03-11T22:19:37.409473+0000","last_became_peered":"2024-03-11T22:19:37.409473+0000","last_unstale":"2024-03-11T22:20:04.803825+0000","last_undegraded":"2024-03-11T22:20:04.803825+0000","last_fullsized":"2024-03-11T22:20:04.803825+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:49:45.588412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807733+0000","last_change":"2024-03-11T22:19:33.224149+0000","last_active":"2024-03-11T22:20:04.807733+0000","last_peered":"2024-03-11T22:20:04.807733+0000","last_clean":"2024-03-11T22:20:04.807733+0000","last_became_active":"2024-03-11T22:19:33.223573+0000","last_became_peered":"2024-03-11T22:19:33.223573+0000","last_unstale":"2024-03-11T22:20:04.807733+0000","last_undegraded":"2024-03-11T22:20:04.807733+0000","last_fullsized":"2024-03-11T22:20:04.807733+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:46:19.321194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807903+0000","last_change":"2024-03-11T22:19:35.399764+0000","last_active":"2024-03-11T22:20:04.807903+0000","last_peered":"2024-03-11T22:20:04.807903+0000","last_clean":"2024-03-11T22:20:04.807903+0000","last_became_active":"2024-03-11T22:19:35.399279+0000","last_became_peered":"2024-03-11T22:19:35.399279+0000","last_unstale":"2024-03-11T22:20:04.807903+0000","last_undegraded":"2024-03-11T22:20:04.807903+0000","last_fullsized":"2024-03-11T22:20:04.807903+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:12:32.851177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806043+0000","last_change":"2024-03-11T22:19:39.423081+0000","last_active":"2024-03-11T22:20:04.806043+0000","last_peered":"2024-03-11T22:20:04.806043+0000","last_clean":"2024-03-11T22:20:04.806043+0000","last_became_active":"2024-03-11T22:19:39.421794+0000","last_became_peered":"2024-03-11T22:19:39.421794+0000","last_unstale":"2024-03-11T22:20:04.806043+0000","last_undegraded":"2024-03-11T22:20:04.806043+0000","last_fullsized":"2024-03-11T22:20:04.806043+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:37:58.954648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.803589+0000","last_change":"2024-03-11T22:19:37.411604+0000","last_active":"2024-03-11T22:20:04.803589+0000","last_peered":"2024-03-11T22:20:04.803589+0000","last_clean":"2024-03-11T22:20:04.803589+0000","last_became_active":"2024-03-11T22:19:37.411212+0000","last_became_peered":"2024-03-11T22:19:37.411212+0000","last_unstale":"2024-03-11T22:20:04.803589+0000","last_undegraded":"2024-03-11T22:20:04.803589+0000","last_fullsized":"2024-03-11T22:20:04.803589+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:27:39.390804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807573+0000","last_change":"2024-03-11T22:19:33.223841+0000","last_active":"2024-03-11T22:20:04.807573+0000","last_peered":"2024-03-11T22:20:04.807573+0000","last_clean":"2024-03-11T22:20:04.807573+0000","last_became_active":"2024-03-11T22:19:33.223411+0000","last_became_peered":"2024-03-11T22:19:33.223411+0000","last_unstale":"2024-03-11T22:20:04.807573+0000","last_undegraded":"2024-03-11T22:20:04.807573+0000","last_fullsized":"2024-03-11T22:20:04.807573+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:20:37.548122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"31'6","reported_seq":32,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806923+0000","last_change":"2024-03-11T22:19:35.408484+0000","last_active":"2024-03-11T22:20:04.806923+0000","last_peered":"2024-03-11T22:20:04.806923+0000","last_clean":"2024-03-11T22:20:04.806923+0000","last_became_active":"2024-03-11T22:19:35.407923+0000","last_became_peered":"2024-03-11T22:19:35.407923+0000","last_unstale":"2024-03-11T22:20:04.806923+0000","last_undegraded":"2024-03-11T22:20:04.806923+0000","last_fullsized":"2024-03-11T22:20:04.806923+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:25:45.235013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805753+0000","last_change":"2024-03-11T22:19:39.425424+0000","last_active":"2024-03-11T22:20:04.805753+0000","last_peered":"2024-03-11T22:20:04.805753+0000","last_clean":"2024-03-11T22:20:04.805753+0000","last_became_active":"2024-03-11T22:19:39.425104+0000","last_became_peered":"2024-03-11T22:19:39.425104+0000","last_unstale":"2024-03-11T22:20:04.805753+0000","last_undegraded":"2024-03-11T22:20:04.805753+0000","last_fullsized":"2024-03-11T22:20:04.805753+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:08:49.681727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807929+0000","last_change":"2024-03-11T22:19:37.412599+0000","last_active":"2024-03-11T22:20:04.807929+0000","last_peered":"2024-03-11T22:20:04.807929+0000","last_clean":"2024-03-11T22:20:04.807929+0000","last_became_active":"2024-03-11T22:19:37.412261+0000","last_became_peered":"2024-03-11T22:19:37.412261+0000","last_unstale":"2024-03-11T22:20:04.807929+0000","last_undegraded":"2024-03-11T22:20:04.807929+0000","last_fullsized":"2024-03-11T22:20:04.807929+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:24:29.453559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808411+0000","last_change":"2024-03-11T22:19:33.224742+0000","last_active":"2024-03-11T22:20:04.808411+0000","last_peered":"2024-03-11T22:20:04.808411+0000","last_clean":"2024-03-11T22:20:04.808411+0000","last_became_active":"2024-03-11T22:19:33.223400+0000","last_became_peered":"2024-03-11T22:19:33.223400+0000","last_unstale":"2024-03-11T22:20:04.808411+0000","last_undegraded":"2024-03-11T22:20:04.808411+0000","last_fullsized":"2024-03-11T22:20:04.808411+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:38:24.671823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808583+0000","last_change":"2024-03-11T22:19:35.402616+0000","last_active":"2024-03-11T22:20:04.808583+0000","last_peered":"2024-03-11T22:20:04.808583+0000","last_clean":"2024-03-11T22:20:04.808583+0000","last_became_active":"2024-03-11T22:19:35.400800+0000","last_became_peered":"2024-03-11T22:19:35.400800+0000","last_unstale":"2024-03-11T22:20:04.808583+0000","last_undegraded":"2024-03-11T22:20:04.808583+0000","last_fullsized":"2024-03-11T22:20:04.808583+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:34:51.564955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806399+0000","last_change":"2024-03-11T22:19:39.423834+0000","last_active":"2024-03-11T22:20:04.806399+0000","last_peered":"2024-03-11T22:20:04.806399+0000","last_clean":"2024-03-11T22:20:04.806399+0000","last_became_active":"2024-03-11T22:19:39.423569+0000","last_became_peered":"2024-03-11T22:19:39.423569+0000","last_unstale":"2024-03-11T22:20:04.806399+0000","last_undegraded":"2024-03-11T22:20:04.806399+0000","last_fullsized":"2024-03-11T22:20:04.806399+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:54:40.591853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804365+0000","last_change":"2024-03-11T22:19:37.409204+0000","last_active":"2024-03-11T22:20:04.804365+0000","last_peered":"2024-03-11T22:20:04.804365+0000","last_clean":"2024-03-11T22:20:04.804365+0000","last_became_active":"2024-03-11T22:19:37.408501+0000","last_became_peered":"2024-03-11T22:19:37.408501+0000","last_unstale":"2024-03-11T22:20:04.804365+0000","last_undegraded":"2024-03-11T22:20:04.804365+0000","last_fullsized":"2024-03-11T22:20:04.804365+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:57:10.752167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807948+0000","last_change":"2024-03-11T22:19:33.223878+0000","last_active":"2024-03-11T22:20:04.807948+0000","last_peered":"2024-03-11T22:20:04.807948+0000","last_clean":"2024-03-11T22:20:04.807948+0000","last_became_active":"2024-03-11T22:19:33.223340+0000","last_became_peered":"2024-03-11T22:19:33.223340+0000","last_unstale":"2024-03-11T22:20:04.807948+0000","last_undegraded":"2024-03-11T22:20:04.807948+0000","last_fullsized":"2024-03-11T22:20:04.807948+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:05:40.023430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808017+0000","last_change":"2024-03-11T22:19:35.402072+0000","last_active":"2024-03-11T22:20:04.808017+0000","last_peered":"2024-03-11T22:20:04.808017+0000","last_clean":"2024-03-11T22:20:04.808017+0000","last_became_active":"2024-03-11T22:19:35.401752+0000","last_became_peered":"2024-03-11T22:19:35.401752+0000","last_unstale":"2024-03-11T22:20:04.808017+0000","last_undegraded":"2024-03-11T22:20:04.808017+0000","last_fullsized":"2024-03-11T22:20:04.808017+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:06:07.665536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805043+0000","last_change":"2024-03-11T22:19:39.425432+0000","last_active":"2024-03-11T22:20:04.805043+0000","last_peered":"2024-03-11T22:20:04.805043+0000","last_clean":"2024-03-11T22:20:04.805043+0000","last_became_active":"2024-03-11T22:19:39.425085+0000","last_became_peered":"2024-03-11T22:19:39.425085+0000","last_unstale":"2024-03-11T22:20:04.805043+0000","last_undegraded":"2024-03-11T22:20:04.805043+0000","last_fullsized":"2024-03-11T22:20:04.805043+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:21:25.202093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807944+0000","last_change":"2024-03-11T22:19:37.408931+0000","last_active":"2024-03-11T22:20:04.807944+0000","last_peered":"2024-03-11T22:20:04.807944+0000","last_clean":"2024-03-11T22:20:04.807944+0000","last_became_active":"2024-03-11T22:19:37.407822+0000","last_became_peered":"2024-03-11T22:19:37.407822+0000","last_unstale":"2024-03-11T22:20:04.807944+0000","last_undegraded":"2024-03-11T22:20:04.807944+0000","last_fullsized":"2024-03-11T22:20:04.807944+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:03:28.270399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807060+0000","last_change":"2024-03-11T22:19:33.222519+0000","last_active":"2024-03-11T22:20:04.807060+0000","last_peered":"2024-03-11T22:20:04.807060+0000","last_clean":"2024-03-11T22:20:04.807060+0000","last_became_active":"2024-03-11T22:19:33.221516+0000","last_became_peered":"2024-03-11T22:19:33.221516+0000","last_unstale":"2024-03-11T22:20:04.807060+0000","last_undegraded":"2024-03-11T22:20:04.807060+0000","last_fullsized":"2024-03-11T22:20:04.807060+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:20:31.979401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804552+0000","last_change":"2024-03-11T22:19:35.405396+0000","last_active":"2024-03-11T22:20:04.804552+0000","last_peered":"2024-03-11T22:20:04.804552+0000","last_clean":"2024-03-11T22:20:04.804552+0000","last_became_active":"2024-03-11T22:19:35.404847+0000","last_became_peered":"2024-03-11T22:19:35.404847+0000","last_unstale":"2024-03-11T22:20:04.804552+0000","last_undegraded":"2024-03-11T22:20:04.804552+0000","last_fullsized":"2024-03-11T22:20:04.804552+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:04:57.935503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805965+0000","last_change":"2024-03-11T22:19:39.424042+0000","last_active":"2024-03-11T22:20:04.805965+0000","last_peered":"2024-03-11T22:20:04.805965+0000","last_clean":"2024-03-11T22:20:04.805965+0000","last_became_active":"2024-03-11T22:19:39.423669+0000","last_became_peered":"2024-03-11T22:19:39.423669+0000","last_unstale":"2024-03-11T22:20:04.805965+0000","last_undegraded":"2024-03-11T22:20:04.805965+0000","last_fullsized":"2024-03-11T22:20:04.805965+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:48:42.502983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807236+0000","last_change":"2024-03-11T22:19:37.406807+0000","last_active":"2024-03-11T22:20:04.807236+0000","last_peered":"2024-03-11T22:20:04.807236+0000","last_clean":"2024-03-11T22:20:04.807236+0000","last_became_active":"2024-03-11T22:19:37.406336+0000","last_became_peered":"2024-03-11T22:19:37.406336+0000","last_unstale":"2024-03-11T22:20:04.807236+0000","last_undegraded":"2024-03-11T22:20:04.807236+0000","last_fullsized":"2024-03-11T22:20:04.807236+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:23:31.831538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806038+0000","last_change":"2024-03-11T22:19:33.222270+0000","last_active":"2024-03-11T22:20:04.806038+0000","last_peered":"2024-03-11T22:20:04.806038+0000","last_clean":"2024-03-11T22:20:04.806038+0000","last_became_active":"2024-03-11T22:19:33.220910+0000","last_became_peered":"2024-03-11T22:19:33.220910+0000","last_unstale":"2024-03-11T22:20:04.806038+0000","last_undegraded":"2024-03-11T22:20:04.806038+0000","last_fullsized":"2024-03-11T22:20:04.806038+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:31:08.541080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808246+0000","last_change":"2024-03-11T22:19:35.399196+0000","last_active":"2024-03-11T22:20:04.808246+0000","last_peered":"2024-03-11T22:20:04.808246+0000","last_clean":"2024-03-11T22:20:04.808246+0000","last_became_active":"2024-03-11T22:19:35.398217+0000","last_became_peered":"2024-03-11T22:19:35.398217+0000","last_unstale":"2024-03-11T22:20:04.808246+0000","last_undegraded":"2024-03-11T22:20:04.808246+0000","last_fullsized":"2024-03-11T22:20:04.808246+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:26:40.413928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808313+0000","last_change":"2024-03-11T22:19:39.423120+0000","last_active":"2024-03-11T22:20:04.808313+0000","last_peered":"2024-03-11T22:20:04.808313+0000","last_clean":"2024-03-11T22:20:04.808313+0000","last_became_active":"2024-03-11T22:19:39.422665+0000","last_became_peered":"2024-03-11T22:19:39.422665+0000","last_unstale":"2024-03-11T22:20:04.808313+0000","last_undegraded":"2024-03-11T22:20:04.808313+0000","last_fullsized":"2024-03-11T22:20:04.808313+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:26:32.267460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806588+0000","last_change":"2024-03-11T22:19:37.411272+0000","last_active":"2024-03-11T22:20:04.806588+0000","last_peered":"2024-03-11T22:20:04.806588+0000","last_clean":"2024-03-11T22:20:04.806588+0000","last_became_active":"2024-03-11T22:19:37.410568+0000","last_became_peered":"2024-03-11T22:19:37.410568+0000","last_unstale":"2024-03-11T22:20:04.806588+0000","last_undegraded":"2024-03-11T22:20:04.806588+0000","last_fullsized":"2024-03-11T22:20:04.806588+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:54:26.261349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805676+0000","last_change":"2024-03-11T22:19:33.224654+0000","last_active":"2024-03-11T22:20:04.805676+0000","last_peered":"2024-03-11T22:20:04.805676+0000","last_clean":"2024-03-11T22:20:04.805676+0000","last_became_active":"2024-03-11T22:19:33.223510+0000","last_became_peered":"2024-03-11T22:19:33.223510+0000","last_unstale":"2024-03-11T22:20:04.805676+0000","last_undegraded":"2024-03-11T22:20:04.805676+0000","last_fullsized":"2024-03-11T22:20:04.805676+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:42:08.193613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"31'4","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805713+0000","last_change":"2024-03-11T22:19:35.403773+0000","last_active":"2024-03-11T22:20:04.805713+0000","last_peered":"2024-03-11T22:20:04.805713+0000","last_clean":"2024-03-11T22:20:04.805713+0000","last_became_active":"2024-03-11T22:19:35.403441+0000","last_became_peered":"2024-03-11T22:19:35.403441+0000","last_unstale":"2024-03-11T22:20:04.805713+0000","last_undegraded":"2024-03-11T22:20:04.805713+0000","last_fullsized":"2024-03-11T22:20:04.805713+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:27:23.947297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806484+0000","last_change":"2024-03-11T22:19:39.427541+0000","last_active":"2024-03-11T22:20:04.806484+0000","last_peered":"2024-03-11T22:20:04.806484+0000","last_clean":"2024-03-11T22:20:04.806484+0000","last_became_active":"2024-03-11T22:19:39.426966+0000","last_became_peered":"2024-03-11T22:19:39.426966+0000","last_unstale":"2024-03-11T22:20:04.806484+0000","last_undegraded":"2024-03-11T22:20:04.806484+0000","last_fullsized":"2024-03-11T22:20:04.806484+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:44:30.276190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805897+0000","last_change":"2024-03-11T22:19:37.409646+0000","last_active":"2024-03-11T22:20:04.805897+0000","last_peered":"2024-03-11T22:20:04.805897+0000","last_clean":"2024-03-11T22:20:04.805897+0000","last_became_active":"2024-03-11T22:19:37.409267+0000","last_became_peered":"2024-03-11T22:19:37.409267+0000","last_unstale":"2024-03-11T22:20:04.805897+0000","last_undegraded":"2024-03-11T22:20:04.805897+0000","last_fullsized":"2024-03-11T22:20:04.805897+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:54:07.573652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805250+0000","last_change":"2024-03-11T22:19:33.224119+0000","last_active":"2024-03-11T22:20:04.805250+0000","last_peered":"2024-03-11T22:20:04.805250+0000","last_clean":"2024-03-11T22:20:04.805250+0000","last_became_active":"2024-03-11T22:19:33.223581+0000","last_became_peered":"2024-03-11T22:19:33.223581+0000","last_unstale":"2024-03-11T22:20:04.805250+0000","last_undegraded":"2024-03-11T22:20:04.805250+0000","last_fullsized":"2024-03-11T22:20:04.805250+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:09:40.078003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805691+0000","last_change":"2024-03-11T22:19:35.397399+0000","last_active":"2024-03-11T22:20:04.805691+0000","last_peered":"2024-03-11T22:20:04.805691+0000","last_clean":"2024-03-11T22:20:04.805691+0000","last_became_active":"2024-03-11T22:19:35.397067+0000","last_became_peered":"2024-03-11T22:19:35.397067+0000","last_unstale":"2024-03-11T22:20:04.805691+0000","last_undegraded":"2024-03-11T22:20:04.805691+0000","last_fullsized":"2024-03-11T22:20:04.805691+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:41:50.500038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805622+0000","last_change":"2024-03-11T22:19:39.425251+0000","last_active":"2024-03-11T22:20:04.805622+0000","last_peered":"2024-03-11T22:20:04.805622+0000","last_clean":"2024-03-11T22:20:04.805622+0000","last_became_active":"2024-03-11T22:19:39.424961+0000","last_became_peered":"2024-03-11T22:19:39.424961+0000","last_unstale":"2024-03-11T22:20:04.805622+0000","last_undegraded":"2024-03-11T22:20:04.805622+0000","last_fullsized":"2024-03-11T22:20:04.805622+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:28:18.644374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"24'2","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808081+0000","last_change":"2024-03-11T22:19:33.223633+0000","last_active":"2024-03-11T22:20:04.808081+0000","last_peered":"2024-03-11T22:20:04.808081+0000","last_clean":"2024-03-11T22:20:04.808081+0000","last_became_active":"2024-03-11T22:19:33.220908+0000","last_became_peered":"2024-03-11T22:19:33.220908+0000","last_unstale":"2024-03-11T22:20:04.808081+0000","last_undegraded":"2024-03-11T22:20:04.808081+0000","last_fullsized":"2024-03-11T22:20:04.808081+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:07:19.347197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807542+0000","last_change":"2024-03-11T22:19:35.402172+0000","last_active":"2024-03-11T22:20:04.807542+0000","last_peered":"2024-03-11T22:20:04.807542+0000","last_clean":"2024-03-11T22:20:04.807542+0000","last_became_active":"2024-03-11T22:19:35.401768+0000","last_became_peered":"2024-03-11T22:19:35.401768+0000","last_unstale":"2024-03-11T22:20:04.807542+0000","last_undegraded":"2024-03-11T22:20:04.807542+0000","last_fullsized":"2024-03-11T22:20:04.807542+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:25:56.253955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805327+0000","last_change":"2024-03-11T22:19:37.407414+0000","last_active":"2024-03-11T22:20:04.805327+0000","last_peered":"2024-03-11T22:20:04.805327+0000","last_clean":"2024-03-11T22:20:04.805327+0000","last_became_active":"2024-03-11T22:19:37.406262+0000","last_became_peered":"2024-03-11T22:19:37.406262+0000","last_unstale":"2024-03-11T22:20:04.805327+0000","last_undegraded":"2024-03-11T22:20:04.805327+0000","last_fullsized":"2024-03-11T22:20:04.805327+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:27:53.343109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804209+0000","last_change":"2024-03-11T22:19:39.422421+0000","last_active":"2024-03-11T22:20:04.804209+0000","last_peered":"2024-03-11T22:20:04.804209+0000","last_clean":"2024-03-11T22:20:04.804209+0000","last_became_active":"2024-03-11T22:19:39.420795+0000","last_became_peered":"2024-03-11T22:19:39.420795+0000","last_unstale":"2024-03-11T22:20:04.804209+0000","last_undegraded":"2024-03-11T22:20:04.804209+0000","last_fullsized":"2024-03-11T22:20:04.804209+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:05:50.607543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805207+0000","last_change":"2024-03-11T22:19:33.223778+0000","last_active":"2024-03-11T22:20:04.805207+0000","last_peered":"2024-03-11T22:20:04.805207+0000","last_clean":"2024-03-11T22:20:04.805207+0000","last_became_active":"2024-03-11T22:19:33.223288+0000","last_became_peered":"2024-03-11T22:19:33.223288+0000","last_unstale":"2024-03-11T22:20:04.805207+0000","last_undegraded":"2024-03-11T22:20:04.805207+0000","last_fullsized":"2024-03-11T22:20:04.805207+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:26:36.823005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"31'16","reported_seq":53,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808547+0000","last_change":"2024-03-11T22:19:35.399000+0000","last_active":"2024-03-11T22:20:04.808547+0000","last_peered":"2024-03-11T22:20:04.808547+0000","last_clean":"2024-03-11T22:20:04.808547+0000","last_became_active":"2024-03-11T22:19:35.398455+0000","last_became_peered":"2024-03-11T22:19:35.398455+0000","last_unstale":"2024-03-11T22:20:04.808547+0000","last_undegraded":"2024-03-11T22:20:04.808547+0000","last_fullsized":"2024-03-11T22:20:04.808547+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:20:05.421124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805839+0000","last_change":"2024-03-11T22:19:37.406574+0000","last_active":"2024-03-11T22:20:04.805839+0000","last_peered":"2024-03-11T22:20:04.805839+0000","last_clean":"2024-03-11T22:20:04.805839+0000","last_became_active":"2024-03-11T22:19:37.406292+0000","last_became_peered":"2024-03-11T22:19:37.406292+0000","last_unstale":"2024-03-11T22:20:04.805839+0000","last_undegraded":"2024-03-11T22:20:04.805839+0000","last_fullsized":"2024-03-11T22:20:04.805839+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:25:29.009066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805425+0000","last_change":"2024-03-11T22:19:39.424856+0000","last_active":"2024-03-11T22:20:04.805425+0000","last_peered":"2024-03-11T22:20:04.805425+0000","last_clean":"2024-03-11T22:20:04.805425+0000","last_became_active":"2024-03-11T22:19:39.424493+0000","last_became_peered":"2024-03-11T22:19:39.424493+0000","last_unstale":"2024-03-11T22:20:04.805425+0000","last_undegraded":"2024-03-11T22:20:04.805425+0000","last_fullsized":"2024-03-11T22:20:04.805425+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:39:00.219777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807064+0000","last_change":"2024-03-11T22:19:33.227443+0000","last_active":"2024-03-11T22:20:04.807064+0000","last_peered":"2024-03-11T22:20:04.807064+0000","last_clean":"2024-03-11T22:20:04.807064+0000","last_became_active":"2024-03-11T22:19:33.226934+0000","last_became_peered":"2024-03-11T22:19:33.226934+0000","last_unstale":"2024-03-11T22:20:04.807064+0000","last_undegraded":"2024-03-11T22:20:04.807064+0000","last_fullsized":"2024-03-11T22:20:04.807064+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:40:26.319058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"31'19","reported_seq":59,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808766+0000","last_change":"2024-03-11T22:19:35.398913+0000","last_active":"2024-03-11T22:20:04.808766+0000","last_peered":"2024-03-11T22:20:04.808766+0000","last_clean":"2024-03-11T22:20:04.808766+0000","last_became_active":"2024-03-11T22:19:35.398178+0000","last_became_peered":"2024-03-11T22:19:35.398178+0000","last_unstale":"2024-03-11T22:20:04.808766+0000","last_undegraded":"2024-03-11T22:20:04.808766+0000","last_fullsized":"2024-03-11T22:20:04.808766+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:50:32.008748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805090+0000","last_change":"2024-03-11T22:19:37.408887+0000","last_active":"2024-03-11T22:20:04.805090+0000","last_peered":"2024-03-11T22:20:04.805090+0000","last_clean":"2024-03-11T22:20:04.805090+0000","last_became_active":"2024-03-11T22:19:37.408525+0000","last_became_peered":"2024-03-11T22:19:37.408525+0000","last_unstale":"2024-03-11T22:20:04.805090+0000","last_undegraded":"2024-03-11T22:20:04.805090+0000","last_fullsized":"2024-03-11T22:20:04.805090+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:56:09.061743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806560+0000","last_change":"2024-03-11T22:19:39.426311+0000","last_active":"2024-03-11T22:20:04.806560+0000","last_peered":"2024-03-11T22:20:04.806560+0000","last_clean":"2024-03-11T22:20:04.806560+0000","last_became_active":"2024-03-11T22:19:39.425637+0000","last_became_peered":"2024-03-11T22:19:39.425637+0000","last_unstale":"2024-03-11T22:20:04.806560+0000","last_undegraded":"2024-03-11T22:20:04.806560+0000","last_fullsized":"2024-03-11T22:20:04.806560+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:09:50.599867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"31'18","reported_seq":55,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804906+0000","last_change":"2024-03-11T22:19:35.404019+0000","last_active":"2024-03-11T22:20:04.804906+0000","last_peered":"2024-03-11T22:20:04.804906+0000","last_clean":"2024-03-11T22:20:04.804906+0000","last_became_active":"2024-03-11T22:19:35.403728+0000","last_became_peered":"2024-03-11T22:19:35.403728+0000","last_unstale":"2024-03-11T22:20:04.804906+0000","last_undegraded":"2024-03-11T22:20:04.804906+0000","last_fullsized":"2024-03-11T22:20:04.804906+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:56:53.978505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.803996+0000","last_change":"2024-03-11T22:19:33.226180+0000","last_active":"2024-03-11T22:20:04.803996+0000","last_peered":"2024-03-11T22:20:04.803996+0000","last_clean":"2024-03-11T22:20:04.803996+0000","last_became_active":"2024-03-11T22:19:33.224693+0000","last_became_peered":"2024-03-11T22:19:33.224693+0000","last_unstale":"2024-03-11T22:20:04.803996+0000","last_undegraded":"2024-03-11T22:20:04.803996+0000","last_fullsized":"2024-03-11T22:20:04.803996+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:19:43.789295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806879+0000","last_change":"2024-03-11T22:19:37.411973+0000","last_active":"2024-03-11T22:20:04.806879+0000","last_peered":"2024-03-11T22:20:04.806879+0000","last_clean":"2024-03-11T22:20:04.806879+0000","last_became_active":"2024-03-11T22:19:37.411049+0000","last_became_peered":"2024-03-11T22:19:37.411049+0000","last_unstale":"2024-03-11T22:20:04.806879+0000","last_undegraded":"2024-03-11T22:20:04.806879+0000","last_fullsized":"2024-03-11T22:20:04.806879+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:39:49.815872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806891+0000","last_change":"2024-03-11T22:19:39.425456+0000","last_active":"2024-03-11T22:20:04.806891+0000","last_peered":"2024-03-11T22:20:04.806891+0000","last_clean":"2024-03-11T22:20:04.806891+0000","last_became_active":"2024-03-11T22:19:39.425166+0000","last_became_peered":"2024-03-11T22:19:39.425166+0000","last_unstale":"2024-03-11T22:20:04.806891+0000","last_undegraded":"2024-03-11T22:20:04.806891+0000","last_fullsized":"2024-03-11T22:20:04.806891+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:16:27.213278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804627+0000","last_change":"2024-03-11T22:19:35.405448+0000","last_active":"2024-03-11T22:20:04.804627+0000","last_peered":"2024-03-11T22:20:04.804627+0000","last_clean":"2024-03-11T22:20:04.804627+0000","last_became_active":"2024-03-11T22:19:35.404950+0000","last_became_peered":"2024-03-11T22:19:35.404950+0000","last_unstale":"2024-03-11T22:20:04.804627+0000","last_undegraded":"2024-03-11T22:20:04.804627+0000","last_fullsized":"2024-03-11T22:20:04.804627+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:42:05.472414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808391+0000","last_change":"2024-03-11T22:19:33.223971+0000","last_active":"2024-03-11T22:20:04.808391+0000","last_peered":"2024-03-11T22:20:04.808391+0000","last_clean":"2024-03-11T22:20:04.808391+0000","last_became_active":"2024-03-11T22:19:33.223486+0000","last_became_peered":"2024-03-11T22:19:33.223486+0000","last_unstale":"2024-03-11T22:20:04.808391+0000","last_undegraded":"2024-03-11T22:20:04.808391+0000","last_fullsized":"2024-03-11T22:20:04.808391+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:04:19.278268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"32'208","reported_seq":314,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:06.903542+0000","last_change":"2024-03-11T22:19:22.170421+0000","last_active":"2024-03-11T22:20:06.903542+0000","last_peered":"2024-03-11T22:20:06.903542+0000","last_clean":"2024-03-11T22:20:06.903542+0000","last_became_active":"2024-03-11T22:19:22.169258+0000","last_became_peered":"2024-03-11T22:19:22.169258+0000","last_unstale":"2024-03-11T22:20:06.903542+0000","last_undegraded":"2024-03-11T22:20:06.903542+0000","last_fullsized":"2024-03-11T22:20:06.903542+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:21.157145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:21.157145+0000","last_clean_scrub_stamp":"2024-03-11T22:19:21.157145+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:47:13.875314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807090+0000","last_change":"2024-03-11T22:19:37.410108+0000","last_active":"2024-03-11T22:20:04.807090+0000","last_peered":"2024-03-11T22:20:04.807090+0000","last_clean":"2024-03-11T22:20:04.807090+0000","last_became_active":"2024-03-11T22:19:37.408945+0000","last_became_peered":"2024-03-11T22:19:37.408945+0000","last_unstale":"2024-03-11T22:20:04.807090+0000","last_undegraded":"2024-03-11T22:20:04.807090+0000","last_fullsized":"2024-03-11T22:20:04.807090+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:24:07.086794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806535+0000","last_change":"2024-03-11T22:19:39.423266+0000","last_active":"2024-03-11T22:20:04.806535+0000","last_peered":"2024-03-11T22:20:04.806535+0000","last_clean":"2024-03-11T22:20:04.806535+0000","last_became_active":"2024-03-11T22:19:39.422808+0000","last_became_peered":"2024-03-11T22:19:39.422808+0000","last_unstale":"2024-03-11T22:20:04.806535+0000","last_undegraded":"2024-03-11T22:20:04.806535+0000","last_fullsized":"2024-03-11T22:20:04.806535+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:18:38.404495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"31'14","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804261+0000","last_change":"2024-03-11T22:19:35.405827+0000","last_active":"2024-03-11T22:20:04.804261+0000","last_peered":"2024-03-11T22:20:04.804261+0000","last_clean":"2024-03-11T22:20:04.804261+0000","last_became_active":"2024-03-11T22:19:35.404817+0000","last_became_peered":"2024-03-11T22:19:35.404817+0000","last_unstale":"2024-03-11T22:20:04.804261+0000","last_undegraded":"2024-03-11T22:20:04.804261+0000","last_fullsized":"2024-03-11T22:20:04.804261+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:50:09.710470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805625+0000","last_change":"2024-03-11T22:19:33.224327+0000","last_active":"2024-03-11T22:20:04.805625+0000","last_peered":"2024-03-11T22:20:04.805625+0000","last_clean":"2024-03-11T22:20:04.805625+0000","last_became_active":"2024-03-11T22:19:33.223788+0000","last_became_peered":"2024-03-11T22:19:33.223788+0000","last_unstale":"2024-03-11T22:20:04.805625+0000","last_undegraded":"2024-03-11T22:20:04.805625+0000","last_fullsized":"2024-03-11T22:20:04.805625+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:42:34.035112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806982+0000","last_change":"2024-03-11T22:19:37.408924+0000","last_active":"2024-03-11T22:20:04.806982+0000","last_peered":"2024-03-11T22:20:04.806982+0000","last_clean":"2024-03-11T22:20:04.806982+0000","last_became_active":"2024-03-11T22:19:37.408633+0000","last_became_peered":"2024-03-11T22:19:37.408633+0000","last_unstale":"2024-03-11T22:20:04.806982+0000","last_undegraded":"2024-03-11T22:20:04.806982+0000","last_fullsized":"2024-03-11T22:20:04.806982+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:11:21.109762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806556+0000","last_change":"2024-03-11T22:19:39.422263+0000","last_active":"2024-03-11T22:20:04.806556+0000","last_peered":"2024-03-11T22:20:04.806556+0000","last_clean":"2024-03-11T22:20:04.806556+0000","last_became_active":"2024-03-11T22:19:39.421928+0000","last_became_peered":"2024-03-11T22:19:39.421928+0000","last_unstale":"2024-03-11T22:20:04.806556+0000","last_undegraded":"2024-03-11T22:20:04.806556+0000","last_fullsized":"2024-03-11T22:20:04.806556+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:00:38.248496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808241+0000","last_change":"2024-03-11T22:19:33.225222+0000","last_active":"2024-03-11T22:20:04.808241+0000","last_peered":"2024-03-11T22:20:04.808241+0000","last_clean":"2024-03-11T22:20:04.808241+0000","last_became_active":"2024-03-11T22:19:33.224822+0000","last_became_peered":"2024-03-11T22:19:33.224822+0000","last_unstale":"2024-03-11T22:20:04.808241+0000","last_undegraded":"2024-03-11T22:20:04.808241+0000","last_fullsized":"2024-03-11T22:20:04.808241+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:22:51.401811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'28","reported_seq":79,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805479+0000","last_change":"2024-03-11T22:19:35.406510+0000","last_active":"2024-03-11T22:20:04.805479+0000","last_peered":"2024-03-11T22:20:04.805479+0000","last_clean":"2024-03-11T22:20:04.805479+0000","last_became_active":"2024-03-11T22:19:35.406147+0000","last_became_peered":"2024-03-11T22:19:35.406147+0000","last_unstale":"2024-03-11T22:20:04.805479+0000","last_undegraded":"2024-03-11T22:20:04.805479+0000","last_fullsized":"2024-03-11T22:20:04.805479+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:00:51.284266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806712+0000","last_change":"2024-03-11T22:19:37.407638+0000","last_active":"2024-03-11T22:20:04.806712+0000","last_peered":"2024-03-11T22:20:04.806712+0000","last_clean":"2024-03-11T22:20:04.806712+0000","last_became_active":"2024-03-11T22:19:37.407139+0000","last_became_peered":"2024-03-11T22:19:37.407139+0000","last_unstale":"2024-03-11T22:20:04.806712+0000","last_undegraded":"2024-03-11T22:20:04.806712+0000","last_fullsized":"2024-03-11T22:20:04.806712+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:39:52.075587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805169+0000","last_change":"2024-03-11T22:19:39.422730+0000","last_active":"2024-03-11T22:20:04.805169+0000","last_peered":"2024-03-11T22:20:04.805169+0000","last_clean":"2024-03-11T22:20:04.805169+0000","last_became_active":"2024-03-11T22:19:39.422189+0000","last_became_peered":"2024-03-11T22:19:39.422189+0000","last_unstale":"2024-03-11T22:20:04.805169+0000","last_undegraded":"2024-03-11T22:20:04.805169+0000","last_fullsized":"2024-03-11T22:20:04.805169+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:31:13.086888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806046+0000","last_change":"2024-03-11T22:19:33.223254+0000","last_active":"2024-03-11T22:20:04.806046+0000","last_peered":"2024-03-11T22:20:04.806046+0000","last_clean":"2024-03-11T22:20:04.806046+0000","last_became_active":"2024-03-11T22:19:33.222940+0000","last_became_peered":"2024-03-11T22:19:33.222940+0000","last_unstale":"2024-03-11T22:20:04.806046+0000","last_undegraded":"2024-03-11T22:20:04.806046+0000","last_fullsized":"2024-03-11T22:20:04.806046+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:05:38.799427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"31'13","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807047+0000","last_change":"2024-03-11T22:19:35.408205+0000","last_active":"2024-03-11T22:20:04.807047+0000","last_peered":"2024-03-11T22:20:04.807047+0000","last_clean":"2024-03-11T22:20:04.807047+0000","last_became_active":"2024-03-11T22:19:35.407525+0000","last_became_peered":"2024-03-11T22:19:35.407525+0000","last_unstale":"2024-03-11T22:20:04.807047+0000","last_undegraded":"2024-03-11T22:20:04.807047+0000","last_fullsized":"2024-03-11T22:20:04.807047+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:07:13.514287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804104+0000","last_change":"2024-03-11T22:19:37.411864+0000","last_active":"2024-03-11T22:20:04.804104+0000","last_peered":"2024-03-11T22:20:04.804104+0000","last_clean":"2024-03-11T22:20:04.804104+0000","last_became_active":"2024-03-11T22:19:37.411506+0000","last_became_peered":"2024-03-11T22:19:37.411506+0000","last_unstale":"2024-03-11T22:20:04.804104+0000","last_undegraded":"2024-03-11T22:20:04.804104+0000","last_fullsized":"2024-03-11T22:20:04.804104+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:15:44.528569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805566+0000","last_change":"2024-03-11T22:19:39.425754+0000","last_active":"2024-03-11T22:20:04.805566+0000","last_peered":"2024-03-11T22:20:04.805566+0000","last_clean":"2024-03-11T22:20:04.805566+0000","last_became_active":"2024-03-11T22:19:39.425446+0000","last_became_peered":"2024-03-11T22:19:39.425446+0000","last_unstale":"2024-03-11T22:20:04.805566+0000","last_undegraded":"2024-03-11T22:20:04.805566+0000","last_fullsized":"2024-03-11T22:20:04.805566+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:40:04.635689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807181+0000","last_change":"2024-03-11T22:19:33.225947+0000","last_active":"2024-03-11T22:20:04.807181+0000","last_peered":"2024-03-11T22:20:04.807181+0000","last_clean":"2024-03-11T22:20:04.807181+0000","last_became_active":"2024-03-11T22:19:33.224666+0000","last_became_peered":"2024-03-11T22:19:33.224666+0000","last_unstale":"2024-03-11T22:20:04.807181+0000","last_undegraded":"2024-03-11T22:20:04.807181+0000","last_fullsized":"2024-03-11T22:20:04.807181+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:41:39.838756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"31'12","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806337+0000","last_change":"2024-03-11T22:19:35.407298+0000","last_active":"2024-03-11T22:20:04.806337+0000","last_peered":"2024-03-11T22:20:04.806337+0000","last_clean":"2024-03-11T22:20:04.806337+0000","last_became_active":"2024-03-11T22:19:35.406794+0000","last_became_peered":"2024-03-11T22:19:35.406794+0000","last_unstale":"2024-03-11T22:20:04.806337+0000","last_undegraded":"2024-03-11T22:20:04.806337+0000","last_fullsized":"2024-03-11T22:20:04.806337+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:50:58.046053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805389+0000","last_change":"2024-03-11T22:19:37.407382+0000","last_active":"2024-03-11T22:20:04.805389+0000","last_peered":"2024-03-11T22:20:04.805389+0000","last_clean":"2024-03-11T22:20:04.805389+0000","last_became_active":"2024-03-11T22:19:37.406433+0000","last_became_peered":"2024-03-11T22:19:37.406433+0000","last_unstale":"2024-03-11T22:20:04.805389+0000","last_undegraded":"2024-03-11T22:20:04.805389+0000","last_fullsized":"2024-03-11T22:20:04.805389+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:14:36.744769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806758+0000","last_change":"2024-03-11T22:19:39.421837+0000","last_active":"2024-03-11T22:20:04.806758+0000","last_peered":"2024-03-11T22:20:04.806758+0000","last_clean":"2024-03-11T22:20:04.806758+0000","last_became_active":"2024-03-11T22:19:39.421512+0000","last_became_peered":"2024-03-11T22:19:39.421512+0000","last_unstale":"2024-03-11T22:20:04.806758+0000","last_undegraded":"2024-03-11T22:20:04.806758+0000","last_fullsized":"2024-03-11T22:20:04.806758+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:53:35.566926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806765+0000","last_change":"2024-03-11T22:19:37.410164+0000","last_active":"2024-03-11T22:20:04.806765+0000","last_peered":"2024-03-11T22:20:04.806765+0000","last_clean":"2024-03-11T22:20:04.806765+0000","last_became_active":"2024-03-11T22:19:37.409756+0000","last_became_peered":"2024-03-11T22:19:37.409756+0000","last_unstale":"2024-03-11T22:20:04.806765+0000","last_undegraded":"2024-03-11T22:20:04.806765+0000","last_fullsized":"2024-03-11T22:20:04.806765+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:57:31.183624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805325+0000","last_change":"2024-03-11T22:19:33.226274+0000","last_active":"2024-03-11T22:20:04.805325+0000","last_peered":"2024-03-11T22:20:04.805325+0000","last_clean":"2024-03-11T22:20:04.805325+0000","last_became_active":"2024-03-11T22:19:33.224751+0000","last_became_peered":"2024-03-11T22:19:33.224751+0000","last_unstale":"2024-03-11T22:20:04.805325+0000","last_undegraded":"2024-03-11T22:20:04.805325+0000","last_fullsized":"2024-03-11T22:20:04.805325+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:41:49.722755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"31'12","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805997+0000","last_change":"2024-03-11T22:19:35.402080+0000","last_active":"2024-03-11T22:20:04.805997+0000","last_peered":"2024-03-11T22:20:04.805997+0000","last_clean":"2024-03-11T22:20:04.805997+0000","last_became_active":"2024-03-11T22:19:35.401763+0000","last_became_peered":"2024-03-11T22:19:35.401763+0000","last_unstale":"2024-03-11T22:20:04.805997+0000","last_undegraded":"2024-03-11T22:20:04.805997+0000","last_fullsized":"2024-03-11T22:20:04.805997+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:26:47.978200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807142+0000","last_change":"2024-03-11T22:19:39.426516+0000","last_active":"2024-03-11T22:20:04.807142+0000","last_peered":"2024-03-11T22:20:04.807142+0000","last_clean":"2024-03-11T22:20:04.807142+0000","last_became_active":"2024-03-11T22:19:39.426096+0000","last_became_peered":"2024-03-11T22:19:39.426096+0000","last_unstale":"2024-03-11T22:20:04.807142+0000","last_undegraded":"2024-03-11T22:20:04.807142+0000","last_fullsized":"2024-03-11T22:20:04.807142+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:07:26.702197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806606+0000","last_change":"2024-03-11T22:19:37.411554+0000","last_active":"2024-03-11T22:20:04.806606+0000","last_peered":"2024-03-11T22:20:04.806606+0000","last_clean":"2024-03-11T22:20:04.806606+0000","last_became_active":"2024-03-11T22:19:37.410697+0000","last_became_peered":"2024-03-11T22:19:37.410697+0000","last_unstale":"2024-03-11T22:20:04.806606+0000","last_undegraded":"2024-03-11T22:20:04.806606+0000","last_fullsized":"2024-03-11T22:20:04.806606+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:39:17.756209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804116+0000","last_change":"2024-03-11T22:19:33.226031+0000","last_active":"2024-03-11T22:20:04.804116+0000","last_peered":"2024-03-11T22:20:04.804116+0000","last_clean":"2024-03-11T22:20:04.804116+0000","last_became_active":"2024-03-11T22:19:33.224214+0000","last_became_peered":"2024-03-11T22:19:33.224214+0000","last_unstale":"2024-03-11T22:20:04.804116+0000","last_undegraded":"2024-03-11T22:20:04.804116+0000","last_fullsized":"2024-03-11T22:20:04.804116+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:22:18.366753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805346+0000","last_change":"2024-03-11T22:19:35.403855+0000","last_active":"2024-03-11T22:20:04.805346+0000","last_peered":"2024-03-11T22:20:04.805346+0000","last_clean":"2024-03-11T22:20:04.805346+0000","last_became_active":"2024-03-11T22:19:35.403584+0000","last_became_peered":"2024-03-11T22:19:35.403584+0000","last_unstale":"2024-03-11T22:20:04.805346+0000","last_undegraded":"2024-03-11T22:20:04.805346+0000","last_fullsized":"2024-03-11T22:20:04.805346+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:59:30.819839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807752+0000","last_change":"2024-03-11T22:19:39.426690+0000","last_active":"2024-03-11T22:20:04.807752+0000","last_peered":"2024-03-11T22:20:04.807752+0000","last_clean":"2024-03-11T22:20:04.807752+0000","last_became_active":"2024-03-11T22:19:39.426361+0000","last_became_peered":"2024-03-11T22:19:39.426361+0000","last_unstale":"2024-03-11T22:20:04.807752+0000","last_undegraded":"2024-03-11T22:20:04.807752+0000","last_fullsized":"2024-03-11T22:20:04.807752+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:24:13.053518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806343+0000","last_change":"2024-03-11T22:19:33.223728+0000","last_active":"2024-03-11T22:20:04.806343+0000","last_peered":"2024-03-11T22:20:04.806343+0000","last_clean":"2024-03-11T22:20:04.806343+0000","last_became_active":"2024-03-11T22:19:33.221867+0000","last_became_peered":"2024-03-11T22:19:33.221867+0000","last_unstale":"2024-03-11T22:20:04.806343+0000","last_undegraded":"2024-03-11T22:20:04.806343+0000","last_fullsized":"2024-03-11T22:20:04.806343+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:53:59.646802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804560+0000","last_change":"2024-03-11T22:19:35.405729+0000","last_active":"2024-03-11T22:20:04.804560+0000","last_peered":"2024-03-11T22:20:04.804560+0000","last_clean":"2024-03-11T22:20:04.804560+0000","last_became_active":"2024-03-11T22:19:35.405292+0000","last_became_peered":"2024-03-11T22:19:35.405292+0000","last_unstale":"2024-03-11T22:20:04.804560+0000","last_undegraded":"2024-03-11T22:20:04.804560+0000","last_fullsized":"2024-03-11T22:20:04.804560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:13:03.428751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807588+0000","last_change":"2024-03-11T22:19:37.412114+0000","last_active":"2024-03-11T22:20:04.807588+0000","last_peered":"2024-03-11T22:20:04.807588+0000","last_clean":"2024-03-11T22:20:04.807588+0000","last_became_active":"2024-03-11T22:19:37.410848+0000","last_became_peered":"2024-03-11T22:19:37.410848+0000","last_unstale":"2024-03-11T22:20:04.807588+0000","last_undegraded":"2024-03-11T22:20:04.807588+0000","last_fullsized":"2024-03-11T22:20:04.807588+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:26:29.905499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804882+0000","last_change":"2024-03-11T22:19:39.421796+0000","last_active":"2024-03-11T22:20:04.804882+0000","last_peered":"2024-03-11T22:20:04.804882+0000","last_clean":"2024-03-11T22:20:04.804882+0000","last_became_active":"2024-03-11T22:19:39.420806+0000","last_became_peered":"2024-03-11T22:19:39.420806+0000","last_unstale":"2024-03-11T22:20:04.804882+0000","last_undegraded":"2024-03-11T22:20:04.804882+0000","last_fullsized":"2024-03-11T22:20:04.804882+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:12:26.556919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806991+0000","last_change":"2024-03-11T22:19:37.410497+0000","last_active":"2024-03-11T22:20:04.806991+0000","last_peered":"2024-03-11T22:20:04.806991+0000","last_clean":"2024-03-11T22:20:04.806991+0000","last_became_active":"2024-03-11T22:19:37.409588+0000","last_became_peered":"2024-03-11T22:19:37.409588+0000","last_unstale":"2024-03-11T22:20:04.806991+0000","last_undegraded":"2024-03-11T22:20:04.806991+0000","last_fullsized":"2024-03-11T22:20:04.806991+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:13:13.305048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806654+0000","last_change":"2024-03-11T22:19:33.223304+0000","last_active":"2024-03-11T22:20:04.806654+0000","last_peered":"2024-03-11T22:20:04.806654+0000","last_clean":"2024-03-11T22:20:04.806654+0000","last_became_active":"2024-03-11T22:19:33.222233+0000","last_became_peered":"2024-03-11T22:19:33.222233+0000","last_unstale":"2024-03-11T22:20:04.806654+0000","last_undegraded":"2024-03-11T22:20:04.806654+0000","last_fullsized":"2024-03-11T22:20:04.806654+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:39.199986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"31'19","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807198+0000","last_change":"2024-03-11T22:19:35.403226+0000","last_active":"2024-03-11T22:20:04.807198+0000","last_peered":"2024-03-11T22:20:04.807198+0000","last_clean":"2024-03-11T22:20:04.807198+0000","last_became_active":"2024-03-11T22:19:35.401996+0000","last_became_peered":"2024-03-11T22:19:35.401996+0000","last_unstale":"2024-03-11T22:20:04.807198+0000","last_undegraded":"2024-03-11T22:20:04.807198+0000","last_fullsized":"2024-03-11T22:20:04.807198+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:32:13.010701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806447+0000","last_change":"2024-03-11T22:19:39.425341+0000","last_active":"2024-03-11T22:20:04.806447+0000","last_peered":"2024-03-11T22:20:04.806447+0000","last_clean":"2024-03-11T22:20:04.806447+0000","last_became_active":"2024-03-11T22:19:39.425061+0000","last_became_peered":"2024-03-11T22:19:39.425061+0000","last_unstale":"2024-03-11T22:20:04.806447+0000","last_undegraded":"2024-03-11T22:20:04.806447+0000","last_fullsized":"2024-03-11T22:20:04.806447+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:09:21.245988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807373+0000","last_change":"2024-03-11T22:19:33.223282+0000","last_active":"2024-03-11T22:20:04.807373+0000","last_peered":"2024-03-11T22:20:04.807373+0000","last_clean":"2024-03-11T22:20:04.807373+0000","last_became_active":"2024-03-11T22:19:33.221822+0000","last_became_peered":"2024-03-11T22:19:33.221822+0000","last_unstale":"2024-03-11T22:20:04.807373+0000","last_undegraded":"2024-03-11T22:20:04.807373+0000","last_fullsized":"2024-03-11T22:20:04.807373+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:16:45.731909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'17","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808265+0000","last_change":"2024-03-11T22:19:35.401115+0000","last_active":"2024-03-11T22:20:04.808265+0000","last_peered":"2024-03-11T22:20:04.808265+0000","last_clean":"2024-03-11T22:20:04.808265+0000","last_became_active":"2024-03-11T22:19:35.400512+0000","last_became_peered":"2024-03-11T22:19:35.400512+0000","last_unstale":"2024-03-11T22:20:04.808265+0000","last_undegraded":"2024-03-11T22:20:04.808265+0000","last_fullsized":"2024-03-11T22:20:04.808265+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:11:49.734624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806119+0000","last_change":"2024-03-11T22:19:37.410288+0000","last_active":"2024-03-11T22:20:04.806119+0000","last_peered":"2024-03-11T22:20:04.806119+0000","last_clean":"2024-03-11T22:20:04.806119+0000","last_became_active":"2024-03-11T22:19:37.409549+0000","last_became_peered":"2024-03-11T22:19:37.409549+0000","last_unstale":"2024-03-11T22:20:04.806119+0000","last_undegraded":"2024-03-11T22:20:04.806119+0000","last_fullsized":"2024-03-11T22:20:04.806119+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:56.951947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.803945+0000","last_change":"2024-03-11T22:19:39.421628+0000","last_active":"2024-03-11T22:20:04.803945+0000","last_peered":"2024-03-11T22:20:04.803945+0000","last_clean":"2024-03-11T22:20:04.803945+0000","last_became_active":"2024-03-11T22:19:39.420689+0000","last_became_peered":"2024-03-11T22:19:39.420689+0000","last_unstale":"2024-03-11T22:20:04.803945+0000","last_undegraded":"2024-03-11T22:20:04.803945+0000","last_fullsized":"2024-03-11T22:20:04.803945+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:35:08.151041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808785+0000","last_change":"2024-03-11T22:19:33.225095+0000","last_active":"2024-03-11T22:20:04.808785+0000","last_peered":"2024-03-11T22:20:04.808785+0000","last_clean":"2024-03-11T22:20:04.808785+0000","last_became_active":"2024-03-11T22:19:33.224329+0000","last_became_peered":"2024-03-11T22:19:33.224329+0000","last_unstale":"2024-03-11T22:20:04.808785+0000","last_undegraded":"2024-03-11T22:20:04.808785+0000","last_fullsized":"2024-03-11T22:20:04.808785+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:05:04.343465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808066+0000","last_change":"2024-03-11T22:19:35.403450+0000","last_active":"2024-03-11T22:20:04.808066+0000","last_peered":"2024-03-11T22:20:04.808066+0000","last_clean":"2024-03-11T22:20:04.808066+0000","last_became_active":"2024-03-11T22:19:35.401957+0000","last_became_peered":"2024-03-11T22:19:35.401957+0000","last_unstale":"2024-03-11T22:20:04.808066+0000","last_undegraded":"2024-03-11T22:20:04.808066+0000","last_fullsized":"2024-03-11T22:20:04.808066+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:01:40.868037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804385+0000","last_change":"2024-03-11T22:19:37.411523+0000","last_active":"2024-03-11T22:20:04.804385+0000","last_peered":"2024-03-11T22:20:04.804385+0000","last_clean":"2024-03-11T22:20:04.804385+0000","last_became_active":"2024-03-11T22:19:37.411180+0000","last_became_peered":"2024-03-11T22:19:37.411180+0000","last_unstale":"2024-03-11T22:20:04.804385+0000","last_undegraded":"2024-03-11T22:20:04.804385+0000","last_fullsized":"2024-03-11T22:20:04.804385+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:02:28.110353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805291+0000","last_change":"2024-03-11T22:19:39.422506+0000","last_active":"2024-03-11T22:20:04.805291+0000","last_peered":"2024-03-11T22:20:04.805291+0000","last_clean":"2024-03-11T22:20:04.805291+0000","last_became_active":"2024-03-11T22:19:39.421631+0000","last_became_peered":"2024-03-11T22:19:39.421631+0000","last_unstale":"2024-03-11T22:20:04.805291+0000","last_undegraded":"2024-03-11T22:20:04.805291+0000","last_fullsized":"2024-03-11T22:20:04.805291+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:25:57.029619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806225+0000","last_change":"2024-03-11T22:19:33.224316+0000","last_active":"2024-03-11T22:20:04.806225+0000","last_peered":"2024-03-11T22:20:04.806225+0000","last_clean":"2024-03-11T22:20:04.806225+0000","last_became_active":"2024-03-11T22:19:33.223900+0000","last_became_peered":"2024-03-11T22:19:33.223900+0000","last_unstale":"2024-03-11T22:20:04.806225+0000","last_undegraded":"2024-03-11T22:20:04.806225+0000","last_fullsized":"2024-03-11T22:20:04.806225+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:05:30.529986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807362+0000","last_change":"2024-03-11T22:19:35.400133+0000","last_active":"2024-03-11T22:20:04.807362+0000","last_peered":"2024-03-11T22:20:04.807362+0000","last_clean":"2024-03-11T22:20:04.807362+0000","last_became_active":"2024-03-11T22:19:35.399786+0000","last_became_peered":"2024-03-11T22:19:35.399786+0000","last_unstale":"2024-03-11T22:20:04.807362+0000","last_undegraded":"2024-03-11T22:20:04.807362+0000","last_fullsized":"2024-03-11T22:20:04.807362+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:34:50.369131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805444+0000","last_change":"2024-03-11T22:19:37.412864+0000","last_active":"2024-03-11T22:20:04.805444+0000","last_peered":"2024-03-11T22:20:04.805444+0000","last_clean":"2024-03-11T22:20:04.805444+0000","last_became_active":"2024-03-11T22:19:37.412550+0000","last_became_peered":"2024-03-11T22:19:37.412550+0000","last_unstale":"2024-03-11T22:20:04.805444+0000","last_undegraded":"2024-03-11T22:20:04.805444+0000","last_fullsized":"2024-03-11T22:20:04.805444+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:28:45.008887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411341,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738391,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:20:14.077 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:20:14.345 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:14.463 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:14 smithi027 ceph-mon[25797]: Updating smithi027:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:20:14.464 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:14 smithi027 ceph-mon[25797]: Updating smithi174:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:20:14.464 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:14 smithi027 ceph-mon[25797]: Updating smithi183:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:20:14.464 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:14 smithi027 ceph-mon[25797]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:20:14.464 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:14 smithi027 ceph-mon[25797]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:20:14.464 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:14 smithi027 ceph-mon[25797]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:20:14.464 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:14 smithi027 ceph-mon[25797]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-11T22:20:14.464 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:14 smithi027 ceph-mon[25797]: Updating smithi174:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:20:14.464 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:14 smithi027 ceph-mon[25797]: Updating smithi027:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:20:14.464 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:14 smithi027 ceph-mon[25797]: Updating smithi183:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:20:14.464 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:14 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:14.464 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:14 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:14.465 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:14 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:14.465 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:14 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:14.465 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:14 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:14.465 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:14 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:14.465 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:14 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:14.465 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:14 smithi027 ceph-mon[25797]: Deploying daemon alertmanager.a on smithi027 2024-03-11T22:20:14.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:14 smithi174 ceph-mon[30085]: Updating smithi027:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:20:14.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:14 smithi174 ceph-mon[30085]: Updating smithi174:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:20:14.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:14 smithi174 ceph-mon[30085]: Updating smithi183:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:20:14.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:14 smithi174 ceph-mon[30085]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:20:14.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:14 smithi174 ceph-mon[30085]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:20:14.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:14 smithi174 ceph-mon[30085]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:20:14.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:14 smithi174 ceph-mon[30085]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-11T22:20:14.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:14 smithi174 ceph-mon[30085]: Updating smithi174:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:20:14.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:14 smithi174 ceph-mon[30085]: Updating smithi027:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:20:14.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:14 smithi174 ceph-mon[30085]: Updating smithi183:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:20:14.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:14 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:14.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:14 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:14.506 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:14 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:14.506 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:14 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:14.506 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:14 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:14.506 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:14 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:14.506 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:14 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:14.506 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:14 smithi174 ceph-mon[30085]: Deploying daemon alertmanager.a on smithi027 2024-03-11T22:20:14.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:14 smithi183 ceph-mon[30906]: Updating smithi027:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:20:14.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:14 smithi183 ceph-mon[30906]: Updating smithi174:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:20:14.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:14 smithi183 ceph-mon[30906]: Updating smithi183:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:20:14.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:14 smithi183 ceph-mon[30906]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:20:14.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:14 smithi183 ceph-mon[30906]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:20:14.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:14 smithi183 ceph-mon[30906]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:20:14.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:14 smithi183 ceph-mon[30906]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-11T22:20:14.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:14 smithi183 ceph-mon[30906]: Updating smithi174:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:20:14.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:14 smithi183 ceph-mon[30906]: Updating smithi027:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:20:14.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:14 smithi183 ceph-mon[30906]: Updating smithi183:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:20:14.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:14 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:14.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:14 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:14.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:14 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:14.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:14 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:14.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:14 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:14.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:14 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:14.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:14 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:14.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:14 smithi183 ceph-mon[30906]: Deploying daemon alertmanager.a on smithi027 2024-03-11T22:20:15.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:15 smithi174 ceph-mon[30085]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:20:15.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:15 smithi027 ceph-mon[25797]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:20:15.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:15 smithi183 ceph-mon[30906]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:20:16.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:16 smithi174 ceph-mon[30085]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T22:20:16.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:16 smithi027 ceph-mon[25797]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T22:20:16.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:16 smithi183 ceph-mon[30906]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T22:20:16.627 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:20:16.629 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-11T22:20:18.180 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-03-11T22:20:14.815553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595908,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96636,"kb_used_data":3108,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126532,"statfs":{"total":287972524032,"available":287873568768,"internally_reserved":0,"allocated":3182592,"data_stored":2109498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":95743312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.997952"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806424+0000","last_change":"2024-03-11T22:19:39.421116+0000","last_active":"2024-03-11T22:20:04.806424+0000","last_peered":"2024-03-11T22:20:04.806424+0000","last_clean":"2024-03-11T22:20:04.806424+0000","last_became_active":"2024-03-11T22:19:39.420843+0000","last_became_peered":"2024-03-11T22:19:39.420843+0000","last_unstale":"2024-03-11T22:20:04.806424+0000","last_undegraded":"2024-03-11T22:20:04.806424+0000","last_fullsized":"2024-03-11T22:20:04.806424+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:53:32.203791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805175+0000","last_change":"2024-03-11T22:19:37.406058+0000","last_active":"2024-03-11T22:20:04.805175+0000","last_peered":"2024-03-11T22:20:04.805175+0000","last_clean":"2024-03-11T22:20:04.805175+0000","last_became_active":"2024-03-11T22:19:37.405848+0000","last_became_peered":"2024-03-11T22:19:37.405848+0000","last_unstale":"2024-03-11T22:20:04.805175+0000","last_undegraded":"2024-03-11T22:20:04.805175+0000","last_fullsized":"2024-03-11T22:20:04.805175+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:58:10.467468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806785+0000","last_change":"2024-03-11T22:19:33.222933+0000","last_active":"2024-03-11T22:20:04.806785+0000","last_peered":"2024-03-11T22:20:04.806785+0000","last_clean":"2024-03-11T22:20:04.806785+0000","last_became_active":"2024-03-11T22:19:33.222624+0000","last_became_peered":"2024-03-11T22:19:33.222624+0000","last_unstale":"2024-03-11T22:20:04.806785+0000","last_undegraded":"2024-03-11T22:20:04.806785+0000","last_fullsized":"2024-03-11T22:20:04.806785+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:25:02.265071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806100+0000","last_change":"2024-03-11T22:19:35.407660+0000","last_active":"2024-03-11T22:20:04.806100+0000","last_peered":"2024-03-11T22:20:04.806100+0000","last_clean":"2024-03-11T22:20:04.806100+0000","last_became_active":"2024-03-11T22:19:35.407385+0000","last_became_peered":"2024-03-11T22:19:35.407385+0000","last_unstale":"2024-03-11T22:20:04.806100+0000","last_undegraded":"2024-03-11T22:20:04.806100+0000","last_fullsized":"2024-03-11T22:20:04.806100+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:59:48.788844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804004+0000","last_change":"2024-03-11T22:19:39.421158+0000","last_active":"2024-03-11T22:20:04.804004+0000","last_peered":"2024-03-11T22:20:04.804004+0000","last_clean":"2024-03-11T22:20:04.804004+0000","last_became_active":"2024-03-11T22:19:39.420791+0000","last_became_peered":"2024-03-11T22:19:39.420791+0000","last_unstale":"2024-03-11T22:20:04.804004+0000","last_undegraded":"2024-03-11T22:20:04.804004+0000","last_fullsized":"2024-03-11T22:20:04.804004+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:07:24.235523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805884+0000","last_change":"2024-03-11T22:19:37.411798+0000","last_active":"2024-03-11T22:20:04.805884+0000","last_peered":"2024-03-11T22:20:04.805884+0000","last_clean":"2024-03-11T22:20:04.805884+0000","last_became_active":"2024-03-11T22:19:37.410719+0000","last_became_peered":"2024-03-11T22:19:37.410719+0000","last_unstale":"2024-03-11T22:20:04.805884+0000","last_undegraded":"2024-03-11T22:20:04.805884+0000","last_fullsized":"2024-03-11T22:20:04.805884+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:43:09.846210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806779+0000","last_change":"2024-03-11T22:19:33.223155+0000","last_active":"2024-03-11T22:20:04.806779+0000","last_peered":"2024-03-11T22:20:04.806779+0000","last_clean":"2024-03-11T22:20:04.806779+0000","last_became_active":"2024-03-11T22:19:33.222732+0000","last_became_peered":"2024-03-11T22:19:33.222732+0000","last_unstale":"2024-03-11T22:20:04.806779+0000","last_undegraded":"2024-03-11T22:20:04.806779+0000","last_fullsized":"2024-03-11T22:20:04.806779+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:26:56.930204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806718+0000","last_change":"2024-03-11T22:19:35.408249+0000","last_active":"2024-03-11T22:20:04.806718+0000","last_peered":"2024-03-11T22:20:04.806718+0000","last_clean":"2024-03-11T22:20:04.806718+0000","last_became_active":"2024-03-11T22:19:35.407907+0000","last_became_peered":"2024-03-11T22:19:35.407907+0000","last_unstale":"2024-03-11T22:20:04.806718+0000","last_undegraded":"2024-03-11T22:20:04.806718+0000","last_fullsized":"2024-03-11T22:20:04.806718+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:13:12.861495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805599+0000","last_change":"2024-03-11T22:19:39.423823+0000","last_active":"2024-03-11T22:20:04.805599+0000","last_peered":"2024-03-11T22:20:04.805599+0000","last_clean":"2024-03-11T22:20:04.805599+0000","last_became_active":"2024-03-11T22:19:39.423538+0000","last_became_peered":"2024-03-11T22:19:39.423538+0000","last_unstale":"2024-03-11T22:20:04.805599+0000","last_undegraded":"2024-03-11T22:20:04.805599+0000","last_fullsized":"2024-03-11T22:20:04.805599+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:28:46.921390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807717+0000","last_change":"2024-03-11T22:19:37.412592+0000","last_active":"2024-03-11T22:20:04.807717+0000","last_peered":"2024-03-11T22:20:04.807717+0000","last_clean":"2024-03-11T22:20:04.807717+0000","last_became_active":"2024-03-11T22:19:37.412130+0000","last_became_peered":"2024-03-11T22:19:37.412130+0000","last_unstale":"2024-03-11T22:20:04.807717+0000","last_undegraded":"2024-03-11T22:20:04.807717+0000","last_fullsized":"2024-03-11T22:20:04.807717+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:46:46.519282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808683+0000","last_change":"2024-03-11T22:19:33.224984+0000","last_active":"2024-03-11T22:20:04.808683+0000","last_peered":"2024-03-11T22:20:04.808683+0000","last_clean":"2024-03-11T22:20:04.808683+0000","last_became_active":"2024-03-11T22:19:33.224251+0000","last_became_peered":"2024-03-11T22:19:33.224251+0000","last_unstale":"2024-03-11T22:20:04.808683+0000","last_undegraded":"2024-03-11T22:20:04.808683+0000","last_fullsized":"2024-03-11T22:20:04.808683+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:04:13.898069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808397+0000","last_change":"2024-03-11T22:19:35.403297+0000","last_active":"2024-03-11T22:20:04.808397+0000","last_peered":"2024-03-11T22:20:04.808397+0000","last_clean":"2024-03-11T22:20:04.808397+0000","last_became_active":"2024-03-11T22:19:35.401404+0000","last_became_peered":"2024-03-11T22:19:35.401404+0000","last_unstale":"2024-03-11T22:20:04.808397+0000","last_undegraded":"2024-03-11T22:20:04.808397+0000","last_fullsized":"2024-03-11T22:20:04.808397+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:39.972534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805678+0000","last_change":"2024-03-11T22:19:39.421824+0000","last_active":"2024-03-11T22:20:04.805678+0000","last_peered":"2024-03-11T22:20:04.805678+0000","last_clean":"2024-03-11T22:20:04.805678+0000","last_became_active":"2024-03-11T22:19:39.421402+0000","last_became_peered":"2024-03-11T22:19:39.421402+0000","last_unstale":"2024-03-11T22:20:04.805678+0000","last_undegraded":"2024-03-11T22:20:04.805678+0000","last_fullsized":"2024-03-11T22:20:04.805678+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:01:58.087718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805810+0000","last_change":"2024-03-11T22:19:37.412169+0000","last_active":"2024-03-11T22:20:04.805810+0000","last_peered":"2024-03-11T22:20:04.805810+0000","last_clean":"2024-03-11T22:20:04.805810+0000","last_became_active":"2024-03-11T22:19:37.410836+0000","last_became_peered":"2024-03-11T22:19:37.410836+0000","last_unstale":"2024-03-11T22:20:04.805810+0000","last_undegraded":"2024-03-11T22:20:04.805810+0000","last_fullsized":"2024-03-11T22:20:04.805810+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:30:19.937932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807430+0000","last_change":"2024-03-11T22:19:33.222677+0000","last_active":"2024-03-11T22:20:04.807430+0000","last_peered":"2024-03-11T22:20:04.807430+0000","last_clean":"2024-03-11T22:20:04.807430+0000","last_became_active":"2024-03-11T22:19:33.221752+0000","last_became_peered":"2024-03-11T22:19:33.221752+0000","last_unstale":"2024-03-11T22:20:04.807430+0000","last_undegraded":"2024-03-11T22:20:04.807430+0000","last_fullsized":"2024-03-11T22:20:04.807430+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:11:40.222643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"31'12","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808499+0000","last_change":"2024-03-11T22:19:35.398123+0000","last_active":"2024-03-11T22:20:04.808499+0000","last_peered":"2024-03-11T22:20:04.808499+0000","last_clean":"2024-03-11T22:20:04.808499+0000","last_became_active":"2024-03-11T22:19:35.397830+0000","last_became_peered":"2024-03-11T22:19:35.397830+0000","last_unstale":"2024-03-11T22:20:04.808499+0000","last_undegraded":"2024-03-11T22:20:04.808499+0000","last_fullsized":"2024-03-11T22:20:04.808499+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:56:29.329916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807454+0000","last_change":"2024-03-11T22:19:39.423075+0000","last_active":"2024-03-11T22:20:04.807454+0000","last_peered":"2024-03-11T22:20:04.807454+0000","last_clean":"2024-03-11T22:20:04.807454+0000","last_became_active":"2024-03-11T22:19:39.422424+0000","last_became_peered":"2024-03-11T22:19:39.422424+0000","last_unstale":"2024-03-11T22:20:04.807454+0000","last_undegraded":"2024-03-11T22:20:04.807454+0000","last_fullsized":"2024-03-11T22:20:04.807454+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:51:28.274829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807722+0000","last_change":"2024-03-11T22:19:37.407427+0000","last_active":"2024-03-11T22:20:04.807722+0000","last_peered":"2024-03-11T22:20:04.807722+0000","last_clean":"2024-03-11T22:20:04.807722+0000","last_became_active":"2024-03-11T22:19:37.407104+0000","last_became_peered":"2024-03-11T22:19:37.407104+0000","last_unstale":"2024-03-11T22:20:04.807722+0000","last_undegraded":"2024-03-11T22:20:04.807722+0000","last_fullsized":"2024-03-11T22:20:04.807722+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:34:01.512236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806860+0000","last_change":"2024-03-11T22:19:33.223374+0000","last_active":"2024-03-11T22:20:04.806860+0000","last_peered":"2024-03-11T22:20:04.806860+0000","last_clean":"2024-03-11T22:20:04.806860+0000","last_became_active":"2024-03-11T22:19:33.221665+0000","last_became_peered":"2024-03-11T22:19:33.221665+0000","last_unstale":"2024-03-11T22:20:04.806860+0000","last_undegraded":"2024-03-11T22:20:04.806860+0000","last_fullsized":"2024-03-11T22:20:04.806860+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:48:00.757046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806346+0000","last_change":"2024-03-11T22:19:35.397986+0000","last_active":"2024-03-11T22:20:04.806346+0000","last_peered":"2024-03-11T22:20:04.806346+0000","last_clean":"2024-03-11T22:20:04.806346+0000","last_became_active":"2024-03-11T22:19:35.397623+0000","last_became_peered":"2024-03-11T22:19:35.397623+0000","last_unstale":"2024-03-11T22:20:04.806346+0000","last_undegraded":"2024-03-11T22:20:04.806346+0000","last_fullsized":"2024-03-11T22:20:04.806346+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:35:29.027191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806045+0000","last_change":"2024-03-11T22:19:39.426841+0000","last_active":"2024-03-11T22:20:04.806045+0000","last_peered":"2024-03-11T22:20:04.806045+0000","last_clean":"2024-03-11T22:20:04.806045+0000","last_became_active":"2024-03-11T22:19:39.426369+0000","last_became_peered":"2024-03-11T22:19:39.426369+0000","last_unstale":"2024-03-11T22:20:04.806045+0000","last_undegraded":"2024-03-11T22:20:04.806045+0000","last_fullsized":"2024-03-11T22:20:04.806045+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:13:42.460874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807211+0000","last_change":"2024-03-11T22:19:37.407440+0000","last_active":"2024-03-11T22:20:04.807211+0000","last_peered":"2024-03-11T22:20:04.807211+0000","last_clean":"2024-03-11T22:20:04.807211+0000","last_became_active":"2024-03-11T22:19:37.407049+0000","last_became_peered":"2024-03-11T22:19:37.407049+0000","last_unstale":"2024-03-11T22:20:04.807211+0000","last_undegraded":"2024-03-11T22:20:04.807211+0000","last_fullsized":"2024-03-11T22:20:04.807211+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:22:26.161223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806883+0000","last_change":"2024-03-11T22:19:33.227456+0000","last_active":"2024-03-11T22:20:04.806883+0000","last_peered":"2024-03-11T22:20:04.806883+0000","last_clean":"2024-03-11T22:20:04.806883+0000","last_became_active":"2024-03-11T22:19:33.227147+0000","last_became_peered":"2024-03-11T22:19:33.227147+0000","last_unstale":"2024-03-11T22:20:04.806883+0000","last_undegraded":"2024-03-11T22:20:04.806883+0000","last_fullsized":"2024-03-11T22:20:04.806883+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:43:14.374168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'5","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806741+0000","last_change":"2024-03-11T22:19:35.397709+0000","last_active":"2024-03-11T22:20:04.806741+0000","last_peered":"2024-03-11T22:20:04.806741+0000","last_clean":"2024-03-11T22:20:04.806741+0000","last_became_active":"2024-03-11T22:19:35.397328+0000","last_became_peered":"2024-03-11T22:19:35.397328+0000","last_unstale":"2024-03-11T22:20:04.806741+0000","last_undegraded":"2024-03-11T22:20:04.806741+0000","last_fullsized":"2024-03-11T22:20:04.806741+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:06:21.456714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805494+0000","last_change":"2024-03-11T22:19:39.426413+0000","last_active":"2024-03-11T22:20:04.805494+0000","last_peered":"2024-03-11T22:20:04.805494+0000","last_clean":"2024-03-11T22:20:04.805494+0000","last_became_active":"2024-03-11T22:19:39.425543+0000","last_became_peered":"2024-03-11T22:19:39.425543+0000","last_unstale":"2024-03-11T22:20:04.805494+0000","last_undegraded":"2024-03-11T22:20:04.805494+0000","last_fullsized":"2024-03-11T22:20:04.805494+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:56:37.440502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"31'8","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808025+0000","last_change":"2024-03-11T22:19:37.406900+0000","last_active":"2024-03-11T22:20:04.808025+0000","last_peered":"2024-03-11T22:20:04.808025+0000","last_clean":"2024-03-11T22:20:04.808025+0000","last_became_active":"2024-03-11T22:19:37.406387+0000","last_became_peered":"2024-03-11T22:19:37.406387+0000","last_unstale":"2024-03-11T22:20:04.808025+0000","last_undegraded":"2024-03-11T22:20:04.808025+0000","last_fullsized":"2024-03-11T22:20:04.808025+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:44:42.280262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804220+0000","last_change":"2024-03-11T22:19:33.223048+0000","last_active":"2024-03-11T22:20:04.804220+0000","last_peered":"2024-03-11T22:20:04.804220+0000","last_clean":"2024-03-11T22:20:04.804220+0000","last_became_active":"2024-03-11T22:19:33.222760+0000","last_became_peered":"2024-03-11T22:19:33.222760+0000","last_unstale":"2024-03-11T22:20:04.804220+0000","last_undegraded":"2024-03-11T22:20:04.804220+0000","last_fullsized":"2024-03-11T22:20:04.804220+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:28:40.522004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.803679+0000","last_change":"2024-03-11T22:19:35.406928+0000","last_active":"2024-03-11T22:20:04.803679+0000","last_peered":"2024-03-11T22:20:04.803679+0000","last_clean":"2024-03-11T22:20:04.803679+0000","last_became_active":"2024-03-11T22:19:35.406155+0000","last_became_peered":"2024-03-11T22:19:35.406155+0000","last_unstale":"2024-03-11T22:20:04.803679+0000","last_undegraded":"2024-03-11T22:20:04.803679+0000","last_fullsized":"2024-03-11T22:20:04.803679+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:22:34.927007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806299+0000","last_change":"2024-03-11T22:19:39.423744+0000","last_active":"2024-03-11T22:20:04.806299+0000","last_peered":"2024-03-11T22:20:04.806299+0000","last_clean":"2024-03-11T22:20:04.806299+0000","last_became_active":"2024-03-11T22:19:39.423369+0000","last_became_peered":"2024-03-11T22:19:39.423369+0000","last_unstale":"2024-03-11T22:20:04.806299+0000","last_undegraded":"2024-03-11T22:20:04.806299+0000","last_fullsized":"2024-03-11T22:20:04.806299+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:24:11.705110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805799+0000","last_change":"2024-03-11T22:19:37.409605+0000","last_active":"2024-03-11T22:20:04.805799+0000","last_peered":"2024-03-11T22:20:04.805799+0000","last_clean":"2024-03-11T22:20:04.805799+0000","last_became_active":"2024-03-11T22:19:37.409209+0000","last_became_peered":"2024-03-11T22:19:37.409209+0000","last_unstale":"2024-03-11T22:20:04.805799+0000","last_undegraded":"2024-03-11T22:20:04.805799+0000","last_fullsized":"2024-03-11T22:20:04.805799+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:36:17.210201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805190+0000","last_change":"2024-03-11T22:19:33.224855+0000","last_active":"2024-03-11T22:20:04.805190+0000","last_peered":"2024-03-11T22:20:04.805190+0000","last_clean":"2024-03-11T22:20:04.805190+0000","last_became_active":"2024-03-11T22:19:33.224355+0000","last_became_peered":"2024-03-11T22:19:33.224355+0000","last_unstale":"2024-03-11T22:20:04.805190+0000","last_undegraded":"2024-03-11T22:20:04.805190+0000","last_fullsized":"2024-03-11T22:20:04.805190+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:01:30.076824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.803832+0000","last_change":"2024-03-11T22:19:35.406509+0000","last_active":"2024-03-11T22:20:04.803832+0000","last_peered":"2024-03-11T22:20:04.803832+0000","last_clean":"2024-03-11T22:20:04.803832+0000","last_became_active":"2024-03-11T22:19:35.406114+0000","last_became_peered":"2024-03-11T22:19:35.406114+0000","last_unstale":"2024-03-11T22:20:04.803832+0000","last_undegraded":"2024-03-11T22:20:04.803832+0000","last_fullsized":"2024-03-11T22:20:04.803832+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:17:32.892908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806581+0000","last_change":"2024-03-11T22:19:39.424762+0000","last_active":"2024-03-11T22:20:04.806581+0000","last_peered":"2024-03-11T22:20:04.806581+0000","last_clean":"2024-03-11T22:20:04.806581+0000","last_became_active":"2024-03-11T22:19:39.424493+0000","last_became_peered":"2024-03-11T22:19:39.424493+0000","last_unstale":"2024-03-11T22:20:04.806581+0000","last_undegraded":"2024-03-11T22:20:04.806581+0000","last_fullsized":"2024-03-11T22:20:04.806581+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:42:01.588578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.803825+0000","last_change":"2024-03-11T22:19:37.409872+0000","last_active":"2024-03-11T22:20:04.803825+0000","last_peered":"2024-03-11T22:20:04.803825+0000","last_clean":"2024-03-11T22:20:04.803825+0000","last_became_active":"2024-03-11T22:19:37.409473+0000","last_became_peered":"2024-03-11T22:19:37.409473+0000","last_unstale":"2024-03-11T22:20:04.803825+0000","last_undegraded":"2024-03-11T22:20:04.803825+0000","last_fullsized":"2024-03-11T22:20:04.803825+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:49:45.588412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807733+0000","last_change":"2024-03-11T22:19:33.224149+0000","last_active":"2024-03-11T22:20:04.807733+0000","last_peered":"2024-03-11T22:20:04.807733+0000","last_clean":"2024-03-11T22:20:04.807733+0000","last_became_active":"2024-03-11T22:19:33.223573+0000","last_became_peered":"2024-03-11T22:19:33.223573+0000","last_unstale":"2024-03-11T22:20:04.807733+0000","last_undegraded":"2024-03-11T22:20:04.807733+0000","last_fullsized":"2024-03-11T22:20:04.807733+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:46:19.321194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807903+0000","last_change":"2024-03-11T22:19:35.399764+0000","last_active":"2024-03-11T22:20:04.807903+0000","last_peered":"2024-03-11T22:20:04.807903+0000","last_clean":"2024-03-11T22:20:04.807903+0000","last_became_active":"2024-03-11T22:19:35.399279+0000","last_became_peered":"2024-03-11T22:19:35.399279+0000","last_unstale":"2024-03-11T22:20:04.807903+0000","last_undegraded":"2024-03-11T22:20:04.807903+0000","last_fullsized":"2024-03-11T22:20:04.807903+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:12:32.851177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806043+0000","last_change":"2024-03-11T22:19:39.423081+0000","last_active":"2024-03-11T22:20:04.806043+0000","last_peered":"2024-03-11T22:20:04.806043+0000","last_clean":"2024-03-11T22:20:04.806043+0000","last_became_active":"2024-03-11T22:19:39.421794+0000","last_became_peered":"2024-03-11T22:19:39.421794+0000","last_unstale":"2024-03-11T22:20:04.806043+0000","last_undegraded":"2024-03-11T22:20:04.806043+0000","last_fullsized":"2024-03-11T22:20:04.806043+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:37:58.954648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.803589+0000","last_change":"2024-03-11T22:19:37.411604+0000","last_active":"2024-03-11T22:20:04.803589+0000","last_peered":"2024-03-11T22:20:04.803589+0000","last_clean":"2024-03-11T22:20:04.803589+0000","last_became_active":"2024-03-11T22:19:37.411212+0000","last_became_peered":"2024-03-11T22:19:37.411212+0000","last_unstale":"2024-03-11T22:20:04.803589+0000","last_undegraded":"2024-03-11T22:20:04.803589+0000","last_fullsized":"2024-03-11T22:20:04.803589+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:27:39.390804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807573+0000","last_change":"2024-03-11T22:19:33.223841+0000","last_active":"2024-03-11T22:20:04.807573+0000","last_peered":"2024-03-11T22:20:04.807573+0000","last_clean":"2024-03-11T22:20:04.807573+0000","last_became_active":"2024-03-11T22:19:33.223411+0000","last_became_peered":"2024-03-11T22:19:33.223411+0000","last_unstale":"2024-03-11T22:20:04.807573+0000","last_undegraded":"2024-03-11T22:20:04.807573+0000","last_fullsized":"2024-03-11T22:20:04.807573+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:20:37.548122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"31'6","reported_seq":32,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806923+0000","last_change":"2024-03-11T22:19:35.408484+0000","last_active":"2024-03-11T22:20:04.806923+0000","last_peered":"2024-03-11T22:20:04.806923+0000","last_clean":"2024-03-11T22:20:04.806923+0000","last_became_active":"2024-03-11T22:19:35.407923+0000","last_became_peered":"2024-03-11T22:19:35.407923+0000","last_unstale":"2024-03-11T22:20:04.806923+0000","last_undegraded":"2024-03-11T22:20:04.806923+0000","last_fullsized":"2024-03-11T22:20:04.806923+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:25:45.235013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805753+0000","last_change":"2024-03-11T22:19:39.425424+0000","last_active":"2024-03-11T22:20:04.805753+0000","last_peered":"2024-03-11T22:20:04.805753+0000","last_clean":"2024-03-11T22:20:04.805753+0000","last_became_active":"2024-03-11T22:19:39.425104+0000","last_became_peered":"2024-03-11T22:19:39.425104+0000","last_unstale":"2024-03-11T22:20:04.805753+0000","last_undegraded":"2024-03-11T22:20:04.805753+0000","last_fullsized":"2024-03-11T22:20:04.805753+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:08:49.681727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807929+0000","last_change":"2024-03-11T22:19:37.412599+0000","last_active":"2024-03-11T22:20:04.807929+0000","last_peered":"2024-03-11T22:20:04.807929+0000","last_clean":"2024-03-11T22:20:04.807929+0000","last_became_active":"2024-03-11T22:19:37.412261+0000","last_became_peered":"2024-03-11T22:19:37.412261+0000","last_unstale":"2024-03-11T22:20:04.807929+0000","last_undegraded":"2024-03-11T22:20:04.807929+0000","last_fullsized":"2024-03-11T22:20:04.807929+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:24:29.453559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808411+0000","last_change":"2024-03-11T22:19:33.224742+0000","last_active":"2024-03-11T22:20:04.808411+0000","last_peered":"2024-03-11T22:20:04.808411+0000","last_clean":"2024-03-11T22:20:04.808411+0000","last_became_active":"2024-03-11T22:19:33.223400+0000","last_became_peered":"2024-03-11T22:19:33.223400+0000","last_unstale":"2024-03-11T22:20:04.808411+0000","last_undegraded":"2024-03-11T22:20:04.808411+0000","last_fullsized":"2024-03-11T22:20:04.808411+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:38:24.671823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808583+0000","last_change":"2024-03-11T22:19:35.402616+0000","last_active":"2024-03-11T22:20:04.808583+0000","last_peered":"2024-03-11T22:20:04.808583+0000","last_clean":"2024-03-11T22:20:04.808583+0000","last_became_active":"2024-03-11T22:19:35.400800+0000","last_became_peered":"2024-03-11T22:19:35.400800+0000","last_unstale":"2024-03-11T22:20:04.808583+0000","last_undegraded":"2024-03-11T22:20:04.808583+0000","last_fullsized":"2024-03-11T22:20:04.808583+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:34:51.564955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806399+0000","last_change":"2024-03-11T22:19:39.423834+0000","last_active":"2024-03-11T22:20:04.806399+0000","last_peered":"2024-03-11T22:20:04.806399+0000","last_clean":"2024-03-11T22:20:04.806399+0000","last_became_active":"2024-03-11T22:19:39.423569+0000","last_became_peered":"2024-03-11T22:19:39.423569+0000","last_unstale":"2024-03-11T22:20:04.806399+0000","last_undegraded":"2024-03-11T22:20:04.806399+0000","last_fullsized":"2024-03-11T22:20:04.806399+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:54:40.591853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804365+0000","last_change":"2024-03-11T22:19:37.409204+0000","last_active":"2024-03-11T22:20:04.804365+0000","last_peered":"2024-03-11T22:20:04.804365+0000","last_clean":"2024-03-11T22:20:04.804365+0000","last_became_active":"2024-03-11T22:19:37.408501+0000","last_became_peered":"2024-03-11T22:19:37.408501+0000","last_unstale":"2024-03-11T22:20:04.804365+0000","last_undegraded":"2024-03-11T22:20:04.804365+0000","last_fullsized":"2024-03-11T22:20:04.804365+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:57:10.752167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807948+0000","last_change":"2024-03-11T22:19:33.223878+0000","last_active":"2024-03-11T22:20:04.807948+0000","last_peered":"2024-03-11T22:20:04.807948+0000","last_clean":"2024-03-11T22:20:04.807948+0000","last_became_active":"2024-03-11T22:19:33.223340+0000","last_became_peered":"2024-03-11T22:19:33.223340+0000","last_unstale":"2024-03-11T22:20:04.807948+0000","last_undegraded":"2024-03-11T22:20:04.807948+0000","last_fullsized":"2024-03-11T22:20:04.807948+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:05:40.023430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808017+0000","last_change":"2024-03-11T22:19:35.402072+0000","last_active":"2024-03-11T22:20:04.808017+0000","last_peered":"2024-03-11T22:20:04.808017+0000","last_clean":"2024-03-11T22:20:04.808017+0000","last_became_active":"2024-03-11T22:19:35.401752+0000","last_became_peered":"2024-03-11T22:19:35.401752+0000","last_unstale":"2024-03-11T22:20:04.808017+0000","last_undegraded":"2024-03-11T22:20:04.808017+0000","last_fullsized":"2024-03-11T22:20:04.808017+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:06:07.665536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805043+0000","last_change":"2024-03-11T22:19:39.425432+0000","last_active":"2024-03-11T22:20:04.805043+0000","last_peered":"2024-03-11T22:20:04.805043+0000","last_clean":"2024-03-11T22:20:04.805043+0000","last_became_active":"2024-03-11T22:19:39.425085+0000","last_became_peered":"2024-03-11T22:19:39.425085+0000","last_unstale":"2024-03-11T22:20:04.805043+0000","last_undegraded":"2024-03-11T22:20:04.805043+0000","last_fullsized":"2024-03-11T22:20:04.805043+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:21:25.202093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807944+0000","last_change":"2024-03-11T22:19:37.408931+0000","last_active":"2024-03-11T22:20:04.807944+0000","last_peered":"2024-03-11T22:20:04.807944+0000","last_clean":"2024-03-11T22:20:04.807944+0000","last_became_active":"2024-03-11T22:19:37.407822+0000","last_became_peered":"2024-03-11T22:19:37.407822+0000","last_unstale":"2024-03-11T22:20:04.807944+0000","last_undegraded":"2024-03-11T22:20:04.807944+0000","last_fullsized":"2024-03-11T22:20:04.807944+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:03:28.270399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807060+0000","last_change":"2024-03-11T22:19:33.222519+0000","last_active":"2024-03-11T22:20:04.807060+0000","last_peered":"2024-03-11T22:20:04.807060+0000","last_clean":"2024-03-11T22:20:04.807060+0000","last_became_active":"2024-03-11T22:19:33.221516+0000","last_became_peered":"2024-03-11T22:19:33.221516+0000","last_unstale":"2024-03-11T22:20:04.807060+0000","last_undegraded":"2024-03-11T22:20:04.807060+0000","last_fullsized":"2024-03-11T22:20:04.807060+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:20:31.979401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804552+0000","last_change":"2024-03-11T22:19:35.405396+0000","last_active":"2024-03-11T22:20:04.804552+0000","last_peered":"2024-03-11T22:20:04.804552+0000","last_clean":"2024-03-11T22:20:04.804552+0000","last_became_active":"2024-03-11T22:19:35.404847+0000","last_became_peered":"2024-03-11T22:19:35.404847+0000","last_unstale":"2024-03-11T22:20:04.804552+0000","last_undegraded":"2024-03-11T22:20:04.804552+0000","last_fullsized":"2024-03-11T22:20:04.804552+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:04:57.935503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805965+0000","last_change":"2024-03-11T22:19:39.424042+0000","last_active":"2024-03-11T22:20:04.805965+0000","last_peered":"2024-03-11T22:20:04.805965+0000","last_clean":"2024-03-11T22:20:04.805965+0000","last_became_active":"2024-03-11T22:19:39.423669+0000","last_became_peered":"2024-03-11T22:19:39.423669+0000","last_unstale":"2024-03-11T22:20:04.805965+0000","last_undegraded":"2024-03-11T22:20:04.805965+0000","last_fullsized":"2024-03-11T22:20:04.805965+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:48:42.502983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807236+0000","last_change":"2024-03-11T22:19:37.406807+0000","last_active":"2024-03-11T22:20:04.807236+0000","last_peered":"2024-03-11T22:20:04.807236+0000","last_clean":"2024-03-11T22:20:04.807236+0000","last_became_active":"2024-03-11T22:19:37.406336+0000","last_became_peered":"2024-03-11T22:19:37.406336+0000","last_unstale":"2024-03-11T22:20:04.807236+0000","last_undegraded":"2024-03-11T22:20:04.807236+0000","last_fullsized":"2024-03-11T22:20:04.807236+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:23:31.831538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806038+0000","last_change":"2024-03-11T22:19:33.222270+0000","last_active":"2024-03-11T22:20:04.806038+0000","last_peered":"2024-03-11T22:20:04.806038+0000","last_clean":"2024-03-11T22:20:04.806038+0000","last_became_active":"2024-03-11T22:19:33.220910+0000","last_became_peered":"2024-03-11T22:19:33.220910+0000","last_unstale":"2024-03-11T22:20:04.806038+0000","last_undegraded":"2024-03-11T22:20:04.806038+0000","last_fullsized":"2024-03-11T22:20:04.806038+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:31:08.541080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808246+0000","last_change":"2024-03-11T22:19:35.399196+0000","last_active":"2024-03-11T22:20:04.808246+0000","last_peered":"2024-03-11T22:20:04.808246+0000","last_clean":"2024-03-11T22:20:04.808246+0000","last_became_active":"2024-03-11T22:19:35.398217+0000","last_became_peered":"2024-03-11T22:19:35.398217+0000","last_unstale":"2024-03-11T22:20:04.808246+0000","last_undegraded":"2024-03-11T22:20:04.808246+0000","last_fullsized":"2024-03-11T22:20:04.808246+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:26:40.413928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808313+0000","last_change":"2024-03-11T22:19:39.423120+0000","last_active":"2024-03-11T22:20:04.808313+0000","last_peered":"2024-03-11T22:20:04.808313+0000","last_clean":"2024-03-11T22:20:04.808313+0000","last_became_active":"2024-03-11T22:19:39.422665+0000","last_became_peered":"2024-03-11T22:19:39.422665+0000","last_unstale":"2024-03-11T22:20:04.808313+0000","last_undegraded":"2024-03-11T22:20:04.808313+0000","last_fullsized":"2024-03-11T22:20:04.808313+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:26:32.267460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806588+0000","last_change":"2024-03-11T22:19:37.411272+0000","last_active":"2024-03-11T22:20:04.806588+0000","last_peered":"2024-03-11T22:20:04.806588+0000","last_clean":"2024-03-11T22:20:04.806588+0000","last_became_active":"2024-03-11T22:19:37.410568+0000","last_became_peered":"2024-03-11T22:19:37.410568+0000","last_unstale":"2024-03-11T22:20:04.806588+0000","last_undegraded":"2024-03-11T22:20:04.806588+0000","last_fullsized":"2024-03-11T22:20:04.806588+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:54:26.261349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805676+0000","last_change":"2024-03-11T22:19:33.224654+0000","last_active":"2024-03-11T22:20:04.805676+0000","last_peered":"2024-03-11T22:20:04.805676+0000","last_clean":"2024-03-11T22:20:04.805676+0000","last_became_active":"2024-03-11T22:19:33.223510+0000","last_became_peered":"2024-03-11T22:19:33.223510+0000","last_unstale":"2024-03-11T22:20:04.805676+0000","last_undegraded":"2024-03-11T22:20:04.805676+0000","last_fullsized":"2024-03-11T22:20:04.805676+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:42:08.193613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"31'4","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805713+0000","last_change":"2024-03-11T22:19:35.403773+0000","last_active":"2024-03-11T22:20:04.805713+0000","last_peered":"2024-03-11T22:20:04.805713+0000","last_clean":"2024-03-11T22:20:04.805713+0000","last_became_active":"2024-03-11T22:19:35.403441+0000","last_became_peered":"2024-03-11T22:19:35.403441+0000","last_unstale":"2024-03-11T22:20:04.805713+0000","last_undegraded":"2024-03-11T22:20:04.805713+0000","last_fullsized":"2024-03-11T22:20:04.805713+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:27:23.947297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806484+0000","last_change":"2024-03-11T22:19:39.427541+0000","last_active":"2024-03-11T22:20:04.806484+0000","last_peered":"2024-03-11T22:20:04.806484+0000","last_clean":"2024-03-11T22:20:04.806484+0000","last_became_active":"2024-03-11T22:19:39.426966+0000","last_became_peered":"2024-03-11T22:19:39.426966+0000","last_unstale":"2024-03-11T22:20:04.806484+0000","last_undegraded":"2024-03-11T22:20:04.806484+0000","last_fullsized":"2024-03-11T22:20:04.806484+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:44:30.276190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805897+0000","last_change":"2024-03-11T22:19:37.409646+0000","last_active":"2024-03-11T22:20:04.805897+0000","last_peered":"2024-03-11T22:20:04.805897+0000","last_clean":"2024-03-11T22:20:04.805897+0000","last_became_active":"2024-03-11T22:19:37.409267+0000","last_became_peered":"2024-03-11T22:19:37.409267+0000","last_unstale":"2024-03-11T22:20:04.805897+0000","last_undegraded":"2024-03-11T22:20:04.805897+0000","last_fullsized":"2024-03-11T22:20:04.805897+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:54:07.573652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805250+0000","last_change":"2024-03-11T22:19:33.224119+0000","last_active":"2024-03-11T22:20:04.805250+0000","last_peered":"2024-03-11T22:20:04.805250+0000","last_clean":"2024-03-11T22:20:04.805250+0000","last_became_active":"2024-03-11T22:19:33.223581+0000","last_became_peered":"2024-03-11T22:19:33.223581+0000","last_unstale":"2024-03-11T22:20:04.805250+0000","last_undegraded":"2024-03-11T22:20:04.805250+0000","last_fullsized":"2024-03-11T22:20:04.805250+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:09:40.078003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805691+0000","last_change":"2024-03-11T22:19:35.397399+0000","last_active":"2024-03-11T22:20:04.805691+0000","last_peered":"2024-03-11T22:20:04.805691+0000","last_clean":"2024-03-11T22:20:04.805691+0000","last_became_active":"2024-03-11T22:19:35.397067+0000","last_became_peered":"2024-03-11T22:19:35.397067+0000","last_unstale":"2024-03-11T22:20:04.805691+0000","last_undegraded":"2024-03-11T22:20:04.805691+0000","last_fullsized":"2024-03-11T22:20:04.805691+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:41:50.500038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805622+0000","last_change":"2024-03-11T22:19:39.425251+0000","last_active":"2024-03-11T22:20:04.805622+0000","last_peered":"2024-03-11T22:20:04.805622+0000","last_clean":"2024-03-11T22:20:04.805622+0000","last_became_active":"2024-03-11T22:19:39.424961+0000","last_became_peered":"2024-03-11T22:19:39.424961+0000","last_unstale":"2024-03-11T22:20:04.805622+0000","last_undegraded":"2024-03-11T22:20:04.805622+0000","last_fullsized":"2024-03-11T22:20:04.805622+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:28:18.644374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"24'2","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808081+0000","last_change":"2024-03-11T22:19:33.223633+0000","last_active":"2024-03-11T22:20:04.808081+0000","last_peered":"2024-03-11T22:20:04.808081+0000","last_clean":"2024-03-11T22:20:04.808081+0000","last_became_active":"2024-03-11T22:19:33.220908+0000","last_became_peered":"2024-03-11T22:19:33.220908+0000","last_unstale":"2024-03-11T22:20:04.808081+0000","last_undegraded":"2024-03-11T22:20:04.808081+0000","last_fullsized":"2024-03-11T22:20:04.808081+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:07:19.347197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807542+0000","last_change":"2024-03-11T22:19:35.402172+0000","last_active":"2024-03-11T22:20:04.807542+0000","last_peered":"2024-03-11T22:20:04.807542+0000","last_clean":"2024-03-11T22:20:04.807542+0000","last_became_active":"2024-03-11T22:19:35.401768+0000","last_became_peered":"2024-03-11T22:19:35.401768+0000","last_unstale":"2024-03-11T22:20:04.807542+0000","last_undegraded":"2024-03-11T22:20:04.807542+0000","last_fullsized":"2024-03-11T22:20:04.807542+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:25:56.253955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805327+0000","last_change":"2024-03-11T22:19:37.407414+0000","last_active":"2024-03-11T22:20:04.805327+0000","last_peered":"2024-03-11T22:20:04.805327+0000","last_clean":"2024-03-11T22:20:04.805327+0000","last_became_active":"2024-03-11T22:19:37.406262+0000","last_became_peered":"2024-03-11T22:19:37.406262+0000","last_unstale":"2024-03-11T22:20:04.805327+0000","last_undegraded":"2024-03-11T22:20:04.805327+0000","last_fullsized":"2024-03-11T22:20:04.805327+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:27:53.343109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804209+0000","last_change":"2024-03-11T22:19:39.422421+0000","last_active":"2024-03-11T22:20:04.804209+0000","last_peered":"2024-03-11T22:20:04.804209+0000","last_clean":"2024-03-11T22:20:04.804209+0000","last_became_active":"2024-03-11T22:19:39.420795+0000","last_became_peered":"2024-03-11T22:19:39.420795+0000","last_unstale":"2024-03-11T22:20:04.804209+0000","last_undegraded":"2024-03-11T22:20:04.804209+0000","last_fullsized":"2024-03-11T22:20:04.804209+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:05:50.607543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805207+0000","last_change":"2024-03-11T22:19:33.223778+0000","last_active":"2024-03-11T22:20:04.805207+0000","last_peered":"2024-03-11T22:20:04.805207+0000","last_clean":"2024-03-11T22:20:04.805207+0000","last_became_active":"2024-03-11T22:19:33.223288+0000","last_became_peered":"2024-03-11T22:19:33.223288+0000","last_unstale":"2024-03-11T22:20:04.805207+0000","last_undegraded":"2024-03-11T22:20:04.805207+0000","last_fullsized":"2024-03-11T22:20:04.805207+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:26:36.823005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"31'16","reported_seq":53,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808547+0000","last_change":"2024-03-11T22:19:35.399000+0000","last_active":"2024-03-11T22:20:04.808547+0000","last_peered":"2024-03-11T22:20:04.808547+0000","last_clean":"2024-03-11T22:20:04.808547+0000","last_became_active":"2024-03-11T22:19:35.398455+0000","last_became_peered":"2024-03-11T22:19:35.398455+0000","last_unstale":"2024-03-11T22:20:04.808547+0000","last_undegraded":"2024-03-11T22:20:04.808547+0000","last_fullsized":"2024-03-11T22:20:04.808547+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:20:05.421124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805839+0000","last_change":"2024-03-11T22:19:37.406574+0000","last_active":"2024-03-11T22:20:04.805839+0000","last_peered":"2024-03-11T22:20:04.805839+0000","last_clean":"2024-03-11T22:20:04.805839+0000","last_became_active":"2024-03-11T22:19:37.406292+0000","last_became_peered":"2024-03-11T22:19:37.406292+0000","last_unstale":"2024-03-11T22:20:04.805839+0000","last_undegraded":"2024-03-11T22:20:04.805839+0000","last_fullsized":"2024-03-11T22:20:04.805839+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:25:29.009066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805425+0000","last_change":"2024-03-11T22:19:39.424856+0000","last_active":"2024-03-11T22:20:04.805425+0000","last_peered":"2024-03-11T22:20:04.805425+0000","last_clean":"2024-03-11T22:20:04.805425+0000","last_became_active":"2024-03-11T22:19:39.424493+0000","last_became_peered":"2024-03-11T22:19:39.424493+0000","last_unstale":"2024-03-11T22:20:04.805425+0000","last_undegraded":"2024-03-11T22:20:04.805425+0000","last_fullsized":"2024-03-11T22:20:04.805425+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:39:00.219777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807064+0000","last_change":"2024-03-11T22:19:33.227443+0000","last_active":"2024-03-11T22:20:04.807064+0000","last_peered":"2024-03-11T22:20:04.807064+0000","last_clean":"2024-03-11T22:20:04.807064+0000","last_became_active":"2024-03-11T22:19:33.226934+0000","last_became_peered":"2024-03-11T22:19:33.226934+0000","last_unstale":"2024-03-11T22:20:04.807064+0000","last_undegraded":"2024-03-11T22:20:04.807064+0000","last_fullsized":"2024-03-11T22:20:04.807064+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:40:26.319058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"31'19","reported_seq":59,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808766+0000","last_change":"2024-03-11T22:19:35.398913+0000","last_active":"2024-03-11T22:20:04.808766+0000","last_peered":"2024-03-11T22:20:04.808766+0000","last_clean":"2024-03-11T22:20:04.808766+0000","last_became_active":"2024-03-11T22:19:35.398178+0000","last_became_peered":"2024-03-11T22:19:35.398178+0000","last_unstale":"2024-03-11T22:20:04.808766+0000","last_undegraded":"2024-03-11T22:20:04.808766+0000","last_fullsized":"2024-03-11T22:20:04.808766+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:50:32.008748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805090+0000","last_change":"2024-03-11T22:19:37.408887+0000","last_active":"2024-03-11T22:20:04.805090+0000","last_peered":"2024-03-11T22:20:04.805090+0000","last_clean":"2024-03-11T22:20:04.805090+0000","last_became_active":"2024-03-11T22:19:37.408525+0000","last_became_peered":"2024-03-11T22:19:37.408525+0000","last_unstale":"2024-03-11T22:20:04.805090+0000","last_undegraded":"2024-03-11T22:20:04.805090+0000","last_fullsized":"2024-03-11T22:20:04.805090+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:56:09.061743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806560+0000","last_change":"2024-03-11T22:19:39.426311+0000","last_active":"2024-03-11T22:20:04.806560+0000","last_peered":"2024-03-11T22:20:04.806560+0000","last_clean":"2024-03-11T22:20:04.806560+0000","last_became_active":"2024-03-11T22:19:39.425637+0000","last_became_peered":"2024-03-11T22:19:39.425637+0000","last_unstale":"2024-03-11T22:20:04.806560+0000","last_undegraded":"2024-03-11T22:20:04.806560+0000","last_fullsized":"2024-03-11T22:20:04.806560+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:09:50.599867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"31'18","reported_seq":55,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804906+0000","last_change":"2024-03-11T22:19:35.404019+0000","last_active":"2024-03-11T22:20:04.804906+0000","last_peered":"2024-03-11T22:20:04.804906+0000","last_clean":"2024-03-11T22:20:04.804906+0000","last_became_active":"2024-03-11T22:19:35.403728+0000","last_became_peered":"2024-03-11T22:19:35.403728+0000","last_unstale":"2024-03-11T22:20:04.804906+0000","last_undegraded":"2024-03-11T22:20:04.804906+0000","last_fullsized":"2024-03-11T22:20:04.804906+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:56:53.978505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.803996+0000","last_change":"2024-03-11T22:19:33.226180+0000","last_active":"2024-03-11T22:20:04.803996+0000","last_peered":"2024-03-11T22:20:04.803996+0000","last_clean":"2024-03-11T22:20:04.803996+0000","last_became_active":"2024-03-11T22:19:33.224693+0000","last_became_peered":"2024-03-11T22:19:33.224693+0000","last_unstale":"2024-03-11T22:20:04.803996+0000","last_undegraded":"2024-03-11T22:20:04.803996+0000","last_fullsized":"2024-03-11T22:20:04.803996+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:19:43.789295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806879+0000","last_change":"2024-03-11T22:19:37.411973+0000","last_active":"2024-03-11T22:20:04.806879+0000","last_peered":"2024-03-11T22:20:04.806879+0000","last_clean":"2024-03-11T22:20:04.806879+0000","last_became_active":"2024-03-11T22:19:37.411049+0000","last_became_peered":"2024-03-11T22:19:37.411049+0000","last_unstale":"2024-03-11T22:20:04.806879+0000","last_undegraded":"2024-03-11T22:20:04.806879+0000","last_fullsized":"2024-03-11T22:20:04.806879+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:39:49.815872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806891+0000","last_change":"2024-03-11T22:19:39.425456+0000","last_active":"2024-03-11T22:20:04.806891+0000","last_peered":"2024-03-11T22:20:04.806891+0000","last_clean":"2024-03-11T22:20:04.806891+0000","last_became_active":"2024-03-11T22:19:39.425166+0000","last_became_peered":"2024-03-11T22:19:39.425166+0000","last_unstale":"2024-03-11T22:20:04.806891+0000","last_undegraded":"2024-03-11T22:20:04.806891+0000","last_fullsized":"2024-03-11T22:20:04.806891+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:16:27.213278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804627+0000","last_change":"2024-03-11T22:19:35.405448+0000","last_active":"2024-03-11T22:20:04.804627+0000","last_peered":"2024-03-11T22:20:04.804627+0000","last_clean":"2024-03-11T22:20:04.804627+0000","last_became_active":"2024-03-11T22:19:35.404950+0000","last_became_peered":"2024-03-11T22:19:35.404950+0000","last_unstale":"2024-03-11T22:20:04.804627+0000","last_undegraded":"2024-03-11T22:20:04.804627+0000","last_fullsized":"2024-03-11T22:20:04.804627+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:42:05.472414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808391+0000","last_change":"2024-03-11T22:19:33.223971+0000","last_active":"2024-03-11T22:20:04.808391+0000","last_peered":"2024-03-11T22:20:04.808391+0000","last_clean":"2024-03-11T22:20:04.808391+0000","last_became_active":"2024-03-11T22:19:33.223486+0000","last_became_peered":"2024-03-11T22:19:33.223486+0000","last_unstale":"2024-03-11T22:20:04.808391+0000","last_undegraded":"2024-03-11T22:20:04.808391+0000","last_fullsized":"2024-03-11T22:20:04.808391+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:04:19.278268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"32'208","reported_seq":314,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:06.903542+0000","last_change":"2024-03-11T22:19:22.170421+0000","last_active":"2024-03-11T22:20:06.903542+0000","last_peered":"2024-03-11T22:20:06.903542+0000","last_clean":"2024-03-11T22:20:06.903542+0000","last_became_active":"2024-03-11T22:19:22.169258+0000","last_became_peered":"2024-03-11T22:19:22.169258+0000","last_unstale":"2024-03-11T22:20:06.903542+0000","last_undegraded":"2024-03-11T22:20:06.903542+0000","last_fullsized":"2024-03-11T22:20:06.903542+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:21.157145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:21.157145+0000","last_clean_scrub_stamp":"2024-03-11T22:19:21.157145+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:47:13.875314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807090+0000","last_change":"2024-03-11T22:19:37.410108+0000","last_active":"2024-03-11T22:20:04.807090+0000","last_peered":"2024-03-11T22:20:04.807090+0000","last_clean":"2024-03-11T22:20:04.807090+0000","last_became_active":"2024-03-11T22:19:37.408945+0000","last_became_peered":"2024-03-11T22:19:37.408945+0000","last_unstale":"2024-03-11T22:20:04.807090+0000","last_undegraded":"2024-03-11T22:20:04.807090+0000","last_fullsized":"2024-03-11T22:20:04.807090+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:24:07.086794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806535+0000","last_change":"2024-03-11T22:19:39.423266+0000","last_active":"2024-03-11T22:20:04.806535+0000","last_peered":"2024-03-11T22:20:04.806535+0000","last_clean":"2024-03-11T22:20:04.806535+0000","last_became_active":"2024-03-11T22:19:39.422808+0000","last_became_peered":"2024-03-11T22:19:39.422808+0000","last_unstale":"2024-03-11T22:20:04.806535+0000","last_undegraded":"2024-03-11T22:20:04.806535+0000","last_fullsized":"2024-03-11T22:20:04.806535+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:18:38.404495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"31'14","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804261+0000","last_change":"2024-03-11T22:19:35.405827+0000","last_active":"2024-03-11T22:20:04.804261+0000","last_peered":"2024-03-11T22:20:04.804261+0000","last_clean":"2024-03-11T22:20:04.804261+0000","last_became_active":"2024-03-11T22:19:35.404817+0000","last_became_peered":"2024-03-11T22:19:35.404817+0000","last_unstale":"2024-03-11T22:20:04.804261+0000","last_undegraded":"2024-03-11T22:20:04.804261+0000","last_fullsized":"2024-03-11T22:20:04.804261+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:50:09.710470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805625+0000","last_change":"2024-03-11T22:19:33.224327+0000","last_active":"2024-03-11T22:20:04.805625+0000","last_peered":"2024-03-11T22:20:04.805625+0000","last_clean":"2024-03-11T22:20:04.805625+0000","last_became_active":"2024-03-11T22:19:33.223788+0000","last_became_peered":"2024-03-11T22:19:33.223788+0000","last_unstale":"2024-03-11T22:20:04.805625+0000","last_undegraded":"2024-03-11T22:20:04.805625+0000","last_fullsized":"2024-03-11T22:20:04.805625+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:42:34.035112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806982+0000","last_change":"2024-03-11T22:19:37.408924+0000","last_active":"2024-03-11T22:20:04.806982+0000","last_peered":"2024-03-11T22:20:04.806982+0000","last_clean":"2024-03-11T22:20:04.806982+0000","last_became_active":"2024-03-11T22:19:37.408633+0000","last_became_peered":"2024-03-11T22:19:37.408633+0000","last_unstale":"2024-03-11T22:20:04.806982+0000","last_undegraded":"2024-03-11T22:20:04.806982+0000","last_fullsized":"2024-03-11T22:20:04.806982+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:11:21.109762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806556+0000","last_change":"2024-03-11T22:19:39.422263+0000","last_active":"2024-03-11T22:20:04.806556+0000","last_peered":"2024-03-11T22:20:04.806556+0000","last_clean":"2024-03-11T22:20:04.806556+0000","last_became_active":"2024-03-11T22:19:39.421928+0000","last_became_peered":"2024-03-11T22:19:39.421928+0000","last_unstale":"2024-03-11T22:20:04.806556+0000","last_undegraded":"2024-03-11T22:20:04.806556+0000","last_fullsized":"2024-03-11T22:20:04.806556+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:00:38.248496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808241+0000","last_change":"2024-03-11T22:19:33.225222+0000","last_active":"2024-03-11T22:20:04.808241+0000","last_peered":"2024-03-11T22:20:04.808241+0000","last_clean":"2024-03-11T22:20:04.808241+0000","last_became_active":"2024-03-11T22:19:33.224822+0000","last_became_peered":"2024-03-11T22:19:33.224822+0000","last_unstale":"2024-03-11T22:20:04.808241+0000","last_undegraded":"2024-03-11T22:20:04.808241+0000","last_fullsized":"2024-03-11T22:20:04.808241+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:22:51.401811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'28","reported_seq":79,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805479+0000","last_change":"2024-03-11T22:19:35.406510+0000","last_active":"2024-03-11T22:20:04.805479+0000","last_peered":"2024-03-11T22:20:04.805479+0000","last_clean":"2024-03-11T22:20:04.805479+0000","last_became_active":"2024-03-11T22:19:35.406147+0000","last_became_peered":"2024-03-11T22:19:35.406147+0000","last_unstale":"2024-03-11T22:20:04.805479+0000","last_undegraded":"2024-03-11T22:20:04.805479+0000","last_fullsized":"2024-03-11T22:20:04.805479+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:00:51.284266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806712+0000","last_change":"2024-03-11T22:19:37.407638+0000","last_active":"2024-03-11T22:20:04.806712+0000","last_peered":"2024-03-11T22:20:04.806712+0000","last_clean":"2024-03-11T22:20:04.806712+0000","last_became_active":"2024-03-11T22:19:37.407139+0000","last_became_peered":"2024-03-11T22:19:37.407139+0000","last_unstale":"2024-03-11T22:20:04.806712+0000","last_undegraded":"2024-03-11T22:20:04.806712+0000","last_fullsized":"2024-03-11T22:20:04.806712+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:39:52.075587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805169+0000","last_change":"2024-03-11T22:19:39.422730+0000","last_active":"2024-03-11T22:20:04.805169+0000","last_peered":"2024-03-11T22:20:04.805169+0000","last_clean":"2024-03-11T22:20:04.805169+0000","last_became_active":"2024-03-11T22:19:39.422189+0000","last_became_peered":"2024-03-11T22:19:39.422189+0000","last_unstale":"2024-03-11T22:20:04.805169+0000","last_undegraded":"2024-03-11T22:20:04.805169+0000","last_fullsized":"2024-03-11T22:20:04.805169+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:31:13.086888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806046+0000","last_change":"2024-03-11T22:19:33.223254+0000","last_active":"2024-03-11T22:20:04.806046+0000","last_peered":"2024-03-11T22:20:04.806046+0000","last_clean":"2024-03-11T22:20:04.806046+0000","last_became_active":"2024-03-11T22:19:33.222940+0000","last_became_peered":"2024-03-11T22:19:33.222940+0000","last_unstale":"2024-03-11T22:20:04.806046+0000","last_undegraded":"2024-03-11T22:20:04.806046+0000","last_fullsized":"2024-03-11T22:20:04.806046+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:05:38.799427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"31'13","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807047+0000","last_change":"2024-03-11T22:19:35.408205+0000","last_active":"2024-03-11T22:20:04.807047+0000","last_peered":"2024-03-11T22:20:04.807047+0000","last_clean":"2024-03-11T22:20:04.807047+0000","last_became_active":"2024-03-11T22:19:35.407525+0000","last_became_peered":"2024-03-11T22:19:35.407525+0000","last_unstale":"2024-03-11T22:20:04.807047+0000","last_undegraded":"2024-03-11T22:20:04.807047+0000","last_fullsized":"2024-03-11T22:20:04.807047+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:07:13.514287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804104+0000","last_change":"2024-03-11T22:19:37.411864+0000","last_active":"2024-03-11T22:20:04.804104+0000","last_peered":"2024-03-11T22:20:04.804104+0000","last_clean":"2024-03-11T22:20:04.804104+0000","last_became_active":"2024-03-11T22:19:37.411506+0000","last_became_peered":"2024-03-11T22:19:37.411506+0000","last_unstale":"2024-03-11T22:20:04.804104+0000","last_undegraded":"2024-03-11T22:20:04.804104+0000","last_fullsized":"2024-03-11T22:20:04.804104+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:15:44.528569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805566+0000","last_change":"2024-03-11T22:19:39.425754+0000","last_active":"2024-03-11T22:20:04.805566+0000","last_peered":"2024-03-11T22:20:04.805566+0000","last_clean":"2024-03-11T22:20:04.805566+0000","last_became_active":"2024-03-11T22:19:39.425446+0000","last_became_peered":"2024-03-11T22:19:39.425446+0000","last_unstale":"2024-03-11T22:20:04.805566+0000","last_undegraded":"2024-03-11T22:20:04.805566+0000","last_fullsized":"2024-03-11T22:20:04.805566+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:40:04.635689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807181+0000","last_change":"2024-03-11T22:19:33.225947+0000","last_active":"2024-03-11T22:20:04.807181+0000","last_peered":"2024-03-11T22:20:04.807181+0000","last_clean":"2024-03-11T22:20:04.807181+0000","last_became_active":"2024-03-11T22:19:33.224666+0000","last_became_peered":"2024-03-11T22:19:33.224666+0000","last_unstale":"2024-03-11T22:20:04.807181+0000","last_undegraded":"2024-03-11T22:20:04.807181+0000","last_fullsized":"2024-03-11T22:20:04.807181+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:41:39.838756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"31'12","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806337+0000","last_change":"2024-03-11T22:19:35.407298+0000","last_active":"2024-03-11T22:20:04.806337+0000","last_peered":"2024-03-11T22:20:04.806337+0000","last_clean":"2024-03-11T22:20:04.806337+0000","last_became_active":"2024-03-11T22:19:35.406794+0000","last_became_peered":"2024-03-11T22:19:35.406794+0000","last_unstale":"2024-03-11T22:20:04.806337+0000","last_undegraded":"2024-03-11T22:20:04.806337+0000","last_fullsized":"2024-03-11T22:20:04.806337+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:50:58.046053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805389+0000","last_change":"2024-03-11T22:19:37.407382+0000","last_active":"2024-03-11T22:20:04.805389+0000","last_peered":"2024-03-11T22:20:04.805389+0000","last_clean":"2024-03-11T22:20:04.805389+0000","last_became_active":"2024-03-11T22:19:37.406433+0000","last_became_peered":"2024-03-11T22:19:37.406433+0000","last_unstale":"2024-03-11T22:20:04.805389+0000","last_undegraded":"2024-03-11T22:20:04.805389+0000","last_fullsized":"2024-03-11T22:20:04.805389+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:14:36.744769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806758+0000","last_change":"2024-03-11T22:19:39.421837+0000","last_active":"2024-03-11T22:20:04.806758+0000","last_peered":"2024-03-11T22:20:04.806758+0000","last_clean":"2024-03-11T22:20:04.806758+0000","last_became_active":"2024-03-11T22:19:39.421512+0000","last_became_peered":"2024-03-11T22:19:39.421512+0000","last_unstale":"2024-03-11T22:20:04.806758+0000","last_undegraded":"2024-03-11T22:20:04.806758+0000","last_fullsized":"2024-03-11T22:20:04.806758+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:53:35.566926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806765+0000","last_change":"2024-03-11T22:19:37.410164+0000","last_active":"2024-03-11T22:20:04.806765+0000","last_peered":"2024-03-11T22:20:04.806765+0000","last_clean":"2024-03-11T22:20:04.806765+0000","last_became_active":"2024-03-11T22:19:37.409756+0000","last_became_peered":"2024-03-11T22:19:37.409756+0000","last_unstale":"2024-03-11T22:20:04.806765+0000","last_undegraded":"2024-03-11T22:20:04.806765+0000","last_fullsized":"2024-03-11T22:20:04.806765+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:57:31.183624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805325+0000","last_change":"2024-03-11T22:19:33.226274+0000","last_active":"2024-03-11T22:20:04.805325+0000","last_peered":"2024-03-11T22:20:04.805325+0000","last_clean":"2024-03-11T22:20:04.805325+0000","last_became_active":"2024-03-11T22:19:33.224751+0000","last_became_peered":"2024-03-11T22:19:33.224751+0000","last_unstale":"2024-03-11T22:20:04.805325+0000","last_undegraded":"2024-03-11T22:20:04.805325+0000","last_fullsized":"2024-03-11T22:20:04.805325+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:41:49.722755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"31'12","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805997+0000","last_change":"2024-03-11T22:19:35.402080+0000","last_active":"2024-03-11T22:20:04.805997+0000","last_peered":"2024-03-11T22:20:04.805997+0000","last_clean":"2024-03-11T22:20:04.805997+0000","last_became_active":"2024-03-11T22:19:35.401763+0000","last_became_peered":"2024-03-11T22:19:35.401763+0000","last_unstale":"2024-03-11T22:20:04.805997+0000","last_undegraded":"2024-03-11T22:20:04.805997+0000","last_fullsized":"2024-03-11T22:20:04.805997+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:26:47.978200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807142+0000","last_change":"2024-03-11T22:19:39.426516+0000","last_active":"2024-03-11T22:20:04.807142+0000","last_peered":"2024-03-11T22:20:04.807142+0000","last_clean":"2024-03-11T22:20:04.807142+0000","last_became_active":"2024-03-11T22:19:39.426096+0000","last_became_peered":"2024-03-11T22:19:39.426096+0000","last_unstale":"2024-03-11T22:20:04.807142+0000","last_undegraded":"2024-03-11T22:20:04.807142+0000","last_fullsized":"2024-03-11T22:20:04.807142+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:07:26.702197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806606+0000","last_change":"2024-03-11T22:19:37.411554+0000","last_active":"2024-03-11T22:20:04.806606+0000","last_peered":"2024-03-11T22:20:04.806606+0000","last_clean":"2024-03-11T22:20:04.806606+0000","last_became_active":"2024-03-11T22:19:37.410697+0000","last_became_peered":"2024-03-11T22:19:37.410697+0000","last_unstale":"2024-03-11T22:20:04.806606+0000","last_undegraded":"2024-03-11T22:20:04.806606+0000","last_fullsized":"2024-03-11T22:20:04.806606+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:39:17.756209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804116+0000","last_change":"2024-03-11T22:19:33.226031+0000","last_active":"2024-03-11T22:20:04.804116+0000","last_peered":"2024-03-11T22:20:04.804116+0000","last_clean":"2024-03-11T22:20:04.804116+0000","last_became_active":"2024-03-11T22:19:33.224214+0000","last_became_peered":"2024-03-11T22:19:33.224214+0000","last_unstale":"2024-03-11T22:20:04.804116+0000","last_undegraded":"2024-03-11T22:20:04.804116+0000","last_fullsized":"2024-03-11T22:20:04.804116+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:22:18.366753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805346+0000","last_change":"2024-03-11T22:19:35.403855+0000","last_active":"2024-03-11T22:20:04.805346+0000","last_peered":"2024-03-11T22:20:04.805346+0000","last_clean":"2024-03-11T22:20:04.805346+0000","last_became_active":"2024-03-11T22:19:35.403584+0000","last_became_peered":"2024-03-11T22:19:35.403584+0000","last_unstale":"2024-03-11T22:20:04.805346+0000","last_undegraded":"2024-03-11T22:20:04.805346+0000","last_fullsized":"2024-03-11T22:20:04.805346+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:59:30.819839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807752+0000","last_change":"2024-03-11T22:19:39.426690+0000","last_active":"2024-03-11T22:20:04.807752+0000","last_peered":"2024-03-11T22:20:04.807752+0000","last_clean":"2024-03-11T22:20:04.807752+0000","last_became_active":"2024-03-11T22:19:39.426361+0000","last_became_peered":"2024-03-11T22:19:39.426361+0000","last_unstale":"2024-03-11T22:20:04.807752+0000","last_undegraded":"2024-03-11T22:20:04.807752+0000","last_fullsized":"2024-03-11T22:20:04.807752+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:24:13.053518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806343+0000","last_change":"2024-03-11T22:19:33.223728+0000","last_active":"2024-03-11T22:20:04.806343+0000","last_peered":"2024-03-11T22:20:04.806343+0000","last_clean":"2024-03-11T22:20:04.806343+0000","last_became_active":"2024-03-11T22:19:33.221867+0000","last_became_peered":"2024-03-11T22:19:33.221867+0000","last_unstale":"2024-03-11T22:20:04.806343+0000","last_undegraded":"2024-03-11T22:20:04.806343+0000","last_fullsized":"2024-03-11T22:20:04.806343+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:53:59.646802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804560+0000","last_change":"2024-03-11T22:19:35.405729+0000","last_active":"2024-03-11T22:20:04.804560+0000","last_peered":"2024-03-11T22:20:04.804560+0000","last_clean":"2024-03-11T22:20:04.804560+0000","last_became_active":"2024-03-11T22:19:35.405292+0000","last_became_peered":"2024-03-11T22:19:35.405292+0000","last_unstale":"2024-03-11T22:20:04.804560+0000","last_undegraded":"2024-03-11T22:20:04.804560+0000","last_fullsized":"2024-03-11T22:20:04.804560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:13:03.428751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807588+0000","last_change":"2024-03-11T22:19:37.412114+0000","last_active":"2024-03-11T22:20:04.807588+0000","last_peered":"2024-03-11T22:20:04.807588+0000","last_clean":"2024-03-11T22:20:04.807588+0000","last_became_active":"2024-03-11T22:19:37.410848+0000","last_became_peered":"2024-03-11T22:19:37.410848+0000","last_unstale":"2024-03-11T22:20:04.807588+0000","last_undegraded":"2024-03-11T22:20:04.807588+0000","last_fullsized":"2024-03-11T22:20:04.807588+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:26:29.905499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804882+0000","last_change":"2024-03-11T22:19:39.421796+0000","last_active":"2024-03-11T22:20:04.804882+0000","last_peered":"2024-03-11T22:20:04.804882+0000","last_clean":"2024-03-11T22:20:04.804882+0000","last_became_active":"2024-03-11T22:19:39.420806+0000","last_became_peered":"2024-03-11T22:19:39.420806+0000","last_unstale":"2024-03-11T22:20:04.804882+0000","last_undegraded":"2024-03-11T22:20:04.804882+0000","last_fullsized":"2024-03-11T22:20:04.804882+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:12:26.556919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806991+0000","last_change":"2024-03-11T22:19:37.410497+0000","last_active":"2024-03-11T22:20:04.806991+0000","last_peered":"2024-03-11T22:20:04.806991+0000","last_clean":"2024-03-11T22:20:04.806991+0000","last_became_active":"2024-03-11T22:19:37.409588+0000","last_became_peered":"2024-03-11T22:19:37.409588+0000","last_unstale":"2024-03-11T22:20:04.806991+0000","last_undegraded":"2024-03-11T22:20:04.806991+0000","last_fullsized":"2024-03-11T22:20:04.806991+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:13:13.305048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806654+0000","last_change":"2024-03-11T22:19:33.223304+0000","last_active":"2024-03-11T22:20:04.806654+0000","last_peered":"2024-03-11T22:20:04.806654+0000","last_clean":"2024-03-11T22:20:04.806654+0000","last_became_active":"2024-03-11T22:19:33.222233+0000","last_became_peered":"2024-03-11T22:19:33.222233+0000","last_unstale":"2024-03-11T22:20:04.806654+0000","last_undegraded":"2024-03-11T22:20:04.806654+0000","last_fullsized":"2024-03-11T22:20:04.806654+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:39.199986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"31'19","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807198+0000","last_change":"2024-03-11T22:19:35.403226+0000","last_active":"2024-03-11T22:20:04.807198+0000","last_peered":"2024-03-11T22:20:04.807198+0000","last_clean":"2024-03-11T22:20:04.807198+0000","last_became_active":"2024-03-11T22:19:35.401996+0000","last_became_peered":"2024-03-11T22:19:35.401996+0000","last_unstale":"2024-03-11T22:20:04.807198+0000","last_undegraded":"2024-03-11T22:20:04.807198+0000","last_fullsized":"2024-03-11T22:20:04.807198+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:32:13.010701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806447+0000","last_change":"2024-03-11T22:19:39.425341+0000","last_active":"2024-03-11T22:20:04.806447+0000","last_peered":"2024-03-11T22:20:04.806447+0000","last_clean":"2024-03-11T22:20:04.806447+0000","last_became_active":"2024-03-11T22:19:39.425061+0000","last_became_peered":"2024-03-11T22:19:39.425061+0000","last_unstale":"2024-03-11T22:20:04.806447+0000","last_undegraded":"2024-03-11T22:20:04.806447+0000","last_fullsized":"2024-03-11T22:20:04.806447+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:09:21.245988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807373+0000","last_change":"2024-03-11T22:19:33.223282+0000","last_active":"2024-03-11T22:20:04.807373+0000","last_peered":"2024-03-11T22:20:04.807373+0000","last_clean":"2024-03-11T22:20:04.807373+0000","last_became_active":"2024-03-11T22:19:33.221822+0000","last_became_peered":"2024-03-11T22:19:33.221822+0000","last_unstale":"2024-03-11T22:20:04.807373+0000","last_undegraded":"2024-03-11T22:20:04.807373+0000","last_fullsized":"2024-03-11T22:20:04.807373+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:16:45.731909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'17","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808265+0000","last_change":"2024-03-11T22:19:35.401115+0000","last_active":"2024-03-11T22:20:04.808265+0000","last_peered":"2024-03-11T22:20:04.808265+0000","last_clean":"2024-03-11T22:20:04.808265+0000","last_became_active":"2024-03-11T22:19:35.400512+0000","last_became_peered":"2024-03-11T22:19:35.400512+0000","last_unstale":"2024-03-11T22:20:04.808265+0000","last_undegraded":"2024-03-11T22:20:04.808265+0000","last_fullsized":"2024-03-11T22:20:04.808265+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:11:49.734624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806119+0000","last_change":"2024-03-11T22:19:37.410288+0000","last_active":"2024-03-11T22:20:04.806119+0000","last_peered":"2024-03-11T22:20:04.806119+0000","last_clean":"2024-03-11T22:20:04.806119+0000","last_became_active":"2024-03-11T22:19:37.409549+0000","last_became_peered":"2024-03-11T22:19:37.409549+0000","last_unstale":"2024-03-11T22:20:04.806119+0000","last_undegraded":"2024-03-11T22:20:04.806119+0000","last_fullsized":"2024-03-11T22:20:04.806119+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:56.951947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.803945+0000","last_change":"2024-03-11T22:19:39.421628+0000","last_active":"2024-03-11T22:20:04.803945+0000","last_peered":"2024-03-11T22:20:04.803945+0000","last_clean":"2024-03-11T22:20:04.803945+0000","last_became_active":"2024-03-11T22:19:39.420689+0000","last_became_peered":"2024-03-11T22:19:39.420689+0000","last_unstale":"2024-03-11T22:20:04.803945+0000","last_undegraded":"2024-03-11T22:20:04.803945+0000","last_fullsized":"2024-03-11T22:20:04.803945+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:35:08.151041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808785+0000","last_change":"2024-03-11T22:19:33.225095+0000","last_active":"2024-03-11T22:20:04.808785+0000","last_peered":"2024-03-11T22:20:04.808785+0000","last_clean":"2024-03-11T22:20:04.808785+0000","last_became_active":"2024-03-11T22:19:33.224329+0000","last_became_peered":"2024-03-11T22:19:33.224329+0000","last_unstale":"2024-03-11T22:20:04.808785+0000","last_undegraded":"2024-03-11T22:20:04.808785+0000","last_fullsized":"2024-03-11T22:20:04.808785+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:05:04.343465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.808066+0000","last_change":"2024-03-11T22:19:35.403450+0000","last_active":"2024-03-11T22:20:04.808066+0000","last_peered":"2024-03-11T22:20:04.808066+0000","last_clean":"2024-03-11T22:20:04.808066+0000","last_became_active":"2024-03-11T22:19:35.401957+0000","last_became_peered":"2024-03-11T22:19:35.401957+0000","last_unstale":"2024-03-11T22:20:04.808066+0000","last_undegraded":"2024-03-11T22:20:04.808066+0000","last_fullsized":"2024-03-11T22:20:04.808066+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:01:40.868037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.804385+0000","last_change":"2024-03-11T22:19:37.411523+0000","last_active":"2024-03-11T22:20:04.804385+0000","last_peered":"2024-03-11T22:20:04.804385+0000","last_clean":"2024-03-11T22:20:04.804385+0000","last_became_active":"2024-03-11T22:19:37.411180+0000","last_became_peered":"2024-03-11T22:19:37.411180+0000","last_unstale":"2024-03-11T22:20:04.804385+0000","last_undegraded":"2024-03-11T22:20:04.804385+0000","last_fullsized":"2024-03-11T22:20:04.804385+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:02:28.110353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805291+0000","last_change":"2024-03-11T22:19:39.422506+0000","last_active":"2024-03-11T22:20:04.805291+0000","last_peered":"2024-03-11T22:20:04.805291+0000","last_clean":"2024-03-11T22:20:04.805291+0000","last_became_active":"2024-03-11T22:19:39.421631+0000","last_became_peered":"2024-03-11T22:19:39.421631+0000","last_unstale":"2024-03-11T22:20:04.805291+0000","last_undegraded":"2024-03-11T22:20:04.805291+0000","last_fullsized":"2024-03-11T22:20:04.805291+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:38.403610+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.403610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:25:57.029619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.806225+0000","last_change":"2024-03-11T22:19:33.224316+0000","last_active":"2024-03-11T22:20:04.806225+0000","last_peered":"2024-03-11T22:20:04.806225+0000","last_clean":"2024-03-11T22:20:04.806225+0000","last_became_active":"2024-03-11T22:19:33.223900+0000","last_became_peered":"2024-03-11T22:19:33.223900+0000","last_unstale":"2024-03-11T22:20:04.806225+0000","last_undegraded":"2024-03-11T22:20:04.806225+0000","last_fullsized":"2024-03-11T22:20:04.806225+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:32.204781+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.204781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:05:30.529986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.807362+0000","last_change":"2024-03-11T22:19:35.400133+0000","last_active":"2024-03-11T22:20:04.807362+0000","last_peered":"2024-03-11T22:20:04.807362+0000","last_clean":"2024-03-11T22:20:04.807362+0000","last_became_active":"2024-03-11T22:19:35.399786+0000","last_became_peered":"2024-03-11T22:19:35.399786+0000","last_unstale":"2024-03-11T22:20:04.807362+0000","last_undegraded":"2024-03-11T22:20:04.807362+0000","last_fullsized":"2024-03-11T22:20:04.807362+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:34.380979+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.380979+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:34:50.369131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:20:04.805444+0000","last_change":"2024-03-11T22:19:37.412864+0000","last_active":"2024-03-11T22:20:04.805444+0000","last_peered":"2024-03-11T22:20:04.805444+0000","last_clean":"2024-03-11T22:20:04.805444+0000","last_became_active":"2024-03-11T22:19:37.412550+0000","last_became_peered":"2024-03-11T22:19:37.412550+0000","last_unstale":"2024-03-11T22:20:04.805444+0000","last_undegraded":"2024-03-11T22:20:04.805444+0000","last_fullsized":"2024-03-11T22:20:04.805444+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:36.389596+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.389596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:28:45.008887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411341,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738391,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:20:18.185 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T22:20:18.185 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-11T22:20:18.185 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-11T22:20:18.185 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-11T22:20:18.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:18 smithi027 ceph-mon[25797]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:20:18.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:18 smithi027 ceph-mon[25797]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T22:20:18.454 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:18.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:18 smithi174 ceph-mon[30085]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:20:18.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:18 smithi174 ceph-mon[30085]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T22:20:18.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:18 smithi183 ceph-mon[30906]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:20:18.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:18 smithi183 ceph-mon[30906]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T22:20:20.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:20 smithi174 ceph-mon[30085]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:20:20.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:20 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:20:20.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:20 smithi027 ceph-mon[25797]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:20:20.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:20 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:20:20.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:20 smithi183 ceph-mon[30906]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:20:20.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:20 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:20:20.890 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:20 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: ::ffff:172.21.15.174 - - [11/Mar/2024:22:20:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:20:21.798 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-11T22:20:21.798 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-11T22:20:22.408 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:22 smithi027 ceph-mon[25797]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:20:22.408 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:22 smithi027 ceph-mon[25797]: from='client.? 172.21.15.27:0/1011866953' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T22:20:22.439 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-11T22:20:22.439 INFO:tasks.cephadm:Setup complete, yielding 2024-03-11T22:20:22.440 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T22:20:22.448 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-03-11T22:20:22.449 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-11T22:20:22.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:22 smithi174 ceph-mon[30085]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:20:22.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:22 smithi174 ceph-mon[30085]: from='client.? 172.21.15.27:0/1011866953' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T22:20:22.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:22 smithi183 ceph-mon[30906]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:20:22.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:22 smithi183 ceph-mon[30906]: from='client.? 172.21.15.27:0/1011866953' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T22:20:23.757 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:24.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:24 smithi174 ceph-mon[30085]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:20:24.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:24 smithi027 ceph-mon[25797]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:20:24.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:24 smithi183 ceph-mon[30906]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:20:25.523 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:20:25 smithi027 podman[58217]: 2024-03-11 22:20:25.425600783 +0000 UTC m=+0.425130175 container create 6e10427aa23c71a8769ef30dfd4d6afe821c5653d7773851414cab1412301267 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:20:25.997 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:20:25 smithi027 podman[58217]: 2024-03-11 22:20:25.762027659 +0000 UTC m=+0.761557051 container init 6e10427aa23c71a8769ef30dfd4d6afe821c5653d7773851414cab1412301267 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:20:25.997 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:20:25 smithi027 podman[58217]: 2024-03-11 22:20:25.768244569 +0000 UTC m=+0.767773961 container start 6e10427aa23c71a8769ef30dfd4d6afe821c5653d7773851414cab1412301267 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:20:25.997 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:20:25 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-alertmanager-a[58254]: ts=2024-03-11T22:20:25.781Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-11T22:20:25.998 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:20:25 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-alertmanager-a[58254]: ts=2024-03-11T22:20:25.781Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-11T22:20:25.998 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:20:25 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-alertmanager-a[58254]: ts=2024-03-11T22:20:25.782Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.27 port=9094 2024-03-11T22:20:25.998 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:20:25 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-alertmanager-a[58254]: ts=2024-03-11T22:20:25.784Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-11T22:20:25.998 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:20:25 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-alertmanager-a[58254]: ts=2024-03-11T22:20:25.807Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T22:20:25.998 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:20:25 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-alertmanager-a[58254]: ts=2024-03-11T22:20:25.807Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T22:20:25.998 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:20:25 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-alertmanager-a[58254]: ts=2024-03-11T22:20:25.810Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-11T22:20:25.998 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:20:25 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-alertmanager-a[58254]: ts=2024-03-11T22:20:25.810Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-11T22:20:25.998 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:20:25 smithi027 bash[58217]: 6e10427aa23c71a8769ef30dfd4d6afe821c5653d7773851414cab1412301267 2024-03-11T22:20:25.998 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:20:25 smithi027 systemd[1]: Started Ceph alertmanager.a for c71f9f14-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:20:25.998 INFO:teuthology.orchestra.run.smithi027.stdout:167 167 2024-03-11T22:20:26.272 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:26 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: [11/Mar/2024:22:20:26] ENGINE Bus STOPPING 2024-03-11T22:20:26.273 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:26 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: [11/Mar/2024:22:20:26] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T22:20:26.273 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:26 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: [11/Mar/2024:22:20:26] ENGINE Bus STOPPED 2024-03-11T22:20:26.273 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:26 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: [11/Mar/2024:22:20:26] ENGINE Bus STARTING 2024-03-11T22:20:26.273 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:26 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: [11/Mar/2024:22:20:26] ENGINE Serving on http://:::9283 2024-03-11T22:20:26.273 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:26 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: [11/Mar/2024:22:20:26] ENGINE Bus STARTED 2024-03-11T22:20:26.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:26 smithi027 ceph-mon[25797]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:26.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:26 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:26.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:26 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:26.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:26 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:26.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:26 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:26.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:26 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:26.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:26 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:26.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:26 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:20:26.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:26 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:26.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:26 smithi174 ceph-mon[30085]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:26.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:26 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:26.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:26 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:26.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:26 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:26.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:26 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:26.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:26 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:26.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:26 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:26.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:26 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:20:26.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:26 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:26.595 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- bash -c 'ceph orch status' 2024-03-11T22:20:26.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:26 smithi183 ceph-mon[30906]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:26.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:26 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:26.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:26 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:26.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:26 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:26.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:26 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:26.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:26 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:26.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:26 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:26.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:26 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:20:26.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:26 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:26.846 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:27.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:27 smithi174 ceph-mon[30085]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T22:20:27.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:27 smithi174 ceph-mon[30085]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:20:27.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:27 smithi174 ceph-mon[30085]: Deploying daemon grafana.a on smithi174 2024-03-11T22:20:27.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:27 smithi027 ceph-mon[25797]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T22:20:27.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:27 smithi027 ceph-mon[25797]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:20:27.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:27 smithi027 ceph-mon[25797]: Deploying daemon grafana.a on smithi174 2024-03-11T22:20:27.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:27 smithi183 ceph-mon[30906]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T22:20:27.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:27 smithi183 ceph-mon[30906]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:20:27.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:27 smithi183 ceph-mon[30906]: Deploying daemon grafana.a on smithi174 2024-03-11T22:20:27.942 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:20:27 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-alertmanager-a[58254]: ts=2024-03-11T22:20:27.785Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000934135s 2024-03-11T22:20:28.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:28 smithi174 ceph-mon[30085]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:28.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:28 smithi027 ceph-mon[25797]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:28.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:28 smithi183 ceph-mon[30906]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:28.666 INFO:teuthology.orchestra.run.smithi027.stdout:Backend: cephadm 2024-03-11T22:20:28.666 INFO:teuthology.orchestra.run.smithi027.stdout:Available: Yes 2024-03-11T22:20:28.666 INFO:teuthology.orchestra.run.smithi027.stdout:Paused: No 2024-03-11T22:20:29.524 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-11T22:20:29.779 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:30.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:30 smithi174 ceph-mon[30085]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:30.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:30 smithi174 ceph-mon[30085]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:30.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:30 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:30.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:30 smithi027 ceph-mon[25797]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:30.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:30 smithi027 ceph-mon[25797]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:30.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:30 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:30.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:30 smithi183 ceph-mon[30906]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:30.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:30 smithi183 ceph-mon[30906]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:30.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:30 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:30.862 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:30 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: ::ffff:172.21.15.174 - - [11/Mar/2024:22:20:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:20:31.593 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:20:31.593 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.a smithi027 *:9093,9094 starting - - - - 2024-03-11T22:20:31.593 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.x smithi174 *:8443,8765 running (2m) 24s ago 2m 447M - 19.0.0-1607-g94a53970 a6477a474d71 aea4c8612d5d 2024-03-11T22:20:31.593 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.y smithi027 *:9283,8765 running (4m) 21s ago 4m 477M - 19.0.0-1607-g94a53970 a6477a474d71 bd177f900fda 2024-03-11T22:20:31.593 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.z smithi183 *:8443,8765 running (2m) 25s ago 2m 447M - 19.0.0-1607-g94a53970 a6477a474d71 6342b9a93650 2024-03-11T22:20:31.593 INFO:teuthology.orchestra.run.smithi027.stdout:mon.a smithi027 running (4m) 21s ago 4m 54.0M 2048M 19.0.0-1607-g94a53970 a6477a474d71 77695fc9e008 2024-03-11T22:20:31.593 INFO:teuthology.orchestra.run.smithi027.stdout:mon.b smithi174 running (2m) 24s ago 2m 45.0M 2048M 19.0.0-1607-g94a53970 a6477a474d71 c06c2efb9f9e 2024-03-11T22:20:31.593 INFO:teuthology.orchestra.run.smithi027.stdout:mon.c smithi183 running (2m) 25s ago 2m 43.5M 2048M 19.0.0-1607-g94a53970 a6477a474d71 b32fe2121d6f 2024-03-11T22:20:31.593 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (2m) 21s ago 2m 63.1M 15.6G 19.0.0-1607-g94a53970 a6477a474d71 7536fc2af891 2024-03-11T22:20:31.593 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi174 running (103s) 24s ago 102s 58.9M 15.6G 19.0.0-1607-g94a53970 a6477a474d71 d02318eccc70 2024-03-11T22:20:31.594 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi183 running (78s) 25s ago 78s 52.5M 16.6G 19.0.0-1607-g94a53970 a6477a474d71 047f9f73e76f 2024-03-11T22:20:31.594 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.a smithi174 *:9095 running (35s) 24s ago 35s 23.8M - 2.43.0 a07b618ecd1d 6253f8395a17 2024-03-11T22:20:31.594 INFO:teuthology.orchestra.run.smithi027.stdout:rgw.foo.a smithi027 *:80 running (60s) 21s ago 60s 75.3M - 19.0.0-1607-g94a53970 a6477a474d71 9cb6f13adf41 2024-03-11T22:20:32.224 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-11T22:20:32.474 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:32.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:32 smithi027 ceph-mon[25797]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:32.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:32 smithi183 ceph-mon[30906]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:32.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:32 smithi174 ceph-mon[30085]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:33.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:33 smithi027 ceph-mon[25797]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:33.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:33 smithi183 ceph-mon[30906]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:33.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:33 smithi174 ceph-mon[30085]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:34.326 INFO:teuthology.orchestra.run.smithi027.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-11T22:20:34.326 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager ?:9093,9094 0/1 - 57s smithi027=a;count:1 2024-03-11T22:20:34.326 INFO:teuthology.orchestra.run.smithi027.stdout:grafana ?:3000 0/1 - 55s smithi174=a;count:1 2024-03-11T22:20:34.326 INFO:teuthology.orchestra.run.smithi027.stdout:mgr 3/3 27s ago 2m smithi027=y;smithi174=x;smithi183=z;count:3 2024-03-11T22:20:34.327 INFO:teuthology.orchestra.run.smithi027.stdout:mon 3/3 27s ago 2m smithi027:172.21.15.27=a;smithi174:172.21.15.174=b;smithi183:172.21.15.183=c;count:3 2024-03-11T22:20:34.327 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter ?:9100 0/3 - 60s smithi027=a;smithi174=b;smithi183=c;count:3 2024-03-11T22:20:34.327 INFO:teuthology.orchestra.run.smithi027.stdout:osd 3 27s ago - 2024-03-11T22:20:34.327 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus ?:9095 1/1 27s ago 63s smithi174=a;count:1 2024-03-11T22:20:34.327 INFO:teuthology.orchestra.run.smithi027.stdout:rgw.foo.a ?:80 1/1 23s ago 62s smithi027=foo.a;count:1 2024-03-11T22:20:34.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:34 smithi027 ceph-mon[25797]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:34.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:34 smithi183 ceph-mon[30906]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:34.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:34 smithi174 ceph-mon[30085]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:34.934 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-11T22:20:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:35.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:35 smithi027 ceph-mon[25797]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:35.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:35 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:20:35.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:35 smithi183 ceph-mon[30906]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:35.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:35 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:20:35.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:35 smithi174 ceph-mon[30085]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:35.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:35 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:20:36.012 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:20:35 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-alertmanager-a[58254]: ts=2024-03-11T22:20:35.786Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002283124s 2024-03-11T22:20:36.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:36 smithi027 ceph-mon[25797]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:36.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:36 smithi183 ceph-mon[30906]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:36.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:36 smithi174 ceph-mon[30085]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:36.964 INFO:teuthology.orchestra.run.smithi027.stdout:HOST ADDR LABELS STATUS 2024-03-11T22:20:36.964 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 172.21.15.27 2024-03-11T22:20:36.964 INFO:teuthology.orchestra.run.smithi027.stdout:smithi174 172.21.15.174 2024-03-11T22:20:36.964 INFO:teuthology.orchestra.run.smithi027.stdout:smithi183 172.21.15.183 2024-03-11T22:20:36.964 INFO:teuthology.orchestra.run.smithi027.stdout:3 hosts in cluster 2024-03-11T22:20:37.572 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-11T22:20:37.820 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:38.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:38 smithi027 ceph-mon[25797]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:38.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:38 smithi027 ceph-mon[25797]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:38.556 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:38 smithi174 ceph-mon[30085]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:38.557 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:38 smithi174 ceph-mon[30085]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:38.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:38 smithi183 ceph-mon[30906]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:38.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:38 smithi183 ceph-mon[30906]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:39.677 INFO:teuthology.orchestra.run.smithi027.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-11T22:20:39.677 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5331001C400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-11T22:20:39.678 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 /dev/nvme1n1 ssd Linux_98e758ac8b77779387f6 89.4G Yes 2m ago 2024-03-11T22:20:39.678 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 /dev/nvme2n1 ssd Linux_a94b0ca315d15e9d7182 89.4G Yes 2m ago 2024-03-11T22:20:39.678 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 /dev/nvme3n1 ssd Linux_5971608164b7de7dbd15 89.4G Yes 2m ago 2024-03-11T22:20:39.678 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 /dev/nvme4n1 ssd Linux_d036649a413c9d693ff0 89.4G No 2m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T22:20:39.678 INFO:teuthology.orchestra.run.smithi027.stdout:smithi174 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62050074400BGN 372G No 103s ago Has a FileSystem, LVM detected 2024-03-11T22:20:39.678 INFO:teuthology.orchestra.run.smithi027.stdout:smithi174 /dev/nvme1n1 ssd Linux_55849c08adb796cb573c 89.4G Yes 103s ago 2024-03-11T22:20:39.678 INFO:teuthology.orchestra.run.smithi027.stdout:smithi174 /dev/nvme2n1 ssd Linux_055b9a0c6f0dfd1cb5fe 89.4G Yes 103s ago 2024-03-11T22:20:39.678 INFO:teuthology.orchestra.run.smithi027.stdout:smithi174 /dev/nvme3n1 ssd Linux_4bbd8d8a96512acc5609 89.4G Yes 103s ago 2024-03-11T22:20:39.678 INFO:teuthology.orchestra.run.smithi027.stdout:smithi174 /dev/nvme4n1 ssd Linux_f941cc85995f17bbd413 89.4G No 103s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T22:20:39.678 INFO:teuthology.orchestra.run.smithi027.stdout:smithi183 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500QJ400BGN 372G No 78s ago Has a FileSystem, LVM detected 2024-03-11T22:20:39.678 INFO:teuthology.orchestra.run.smithi027.stdout:smithi183 /dev/nvme1n1 ssd Linux_55f82714da76bf163c37 89.4G Yes 78s ago 2024-03-11T22:20:39.678 INFO:teuthology.orchestra.run.smithi027.stdout:smithi183 /dev/nvme2n1 ssd Linux_1c4319991c9dd16ce489 89.4G Yes 78s ago 2024-03-11T22:20:39.678 INFO:teuthology.orchestra.run.smithi027.stdout:smithi183 /dev/nvme3n1 ssd Linux_048f6294288f52b78793 89.4G Yes 78s ago 2024-03-11T22:20:39.678 INFO:teuthology.orchestra.run.smithi027.stdout:smithi183 /dev/nvme4n1 ssd Linux_96ee08236bd417d0c5f4 89.4G No 78s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T22:20:40.311 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-11T22:20:40.558 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:40.560 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:40 smithi027 ceph-mon[25797]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:40.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:40 smithi183 ceph-mon[30906]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:40.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:40 smithi174 ceph-mon[30085]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:40.912 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:40 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: ::ffff:172.21.15.174 - - [11/Mar/2024:22:20:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:20:41.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:41 smithi183 ceph-mon[30906]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:41.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:41 smithi027 ceph-mon[25797]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:41.755 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:41 smithi174 ceph-mon[30085]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:42.431 INFO:teuthology.orchestra.run.smithi027.stdout:service_type: alertmanager 2024-03-11T22:20:42.431 INFO:teuthology.orchestra.run.smithi027.stdout:service_name: alertmanager 2024-03-11T22:20:42.432 INFO:teuthology.orchestra.run.smithi027.stdout:placement: 2024-03-11T22:20:42.432 INFO:teuthology.orchestra.run.smithi027.stdout: count: 1 2024-03-11T22:20:42.432 INFO:teuthology.orchestra.run.smithi027.stdout: hosts: 2024-03-11T22:20:42.432 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi027=a 2024-03-11T22:20:42.432 INFO:teuthology.orchestra.run.smithi027.stdout:status: 2024-03-11T22:20:42.432 INFO:teuthology.orchestra.run.smithi027.stdout: created: '2024-03-11T22:19:36.371944Z' 2024-03-11T22:20:42.432 INFO:teuthology.orchestra.run.smithi027.stdout: ports: 2024-03-11T22:20:42.432 INFO:teuthology.orchestra.run.smithi027.stdout: - 9093 2024-03-11T22:20:42.432 INFO:teuthology.orchestra.run.smithi027.stdout: - 9094 2024-03-11T22:20:42.432 INFO:teuthology.orchestra.run.smithi027.stdout: running: 0 2024-03-11T22:20:42.432 INFO:teuthology.orchestra.run.smithi027.stdout: size: 1 2024-03-11T22:20:42.432 INFO:teuthology.orchestra.run.smithi027.stdout:events: 2024-03-11T22:20:42.432 INFO:teuthology.orchestra.run.smithi027.stdout:- 2024-03-11T22:20:25.947117Z service:alertmanager [INFO] "service was created" 2024-03-11T22:20:42.432 INFO:teuthology.orchestra.run.smithi027.stdout:--- 2024-03-11T22:20:42.432 INFO:teuthology.orchestra.run.smithi027.stdout:service_type: grafana 2024-03-11T22:20:42.432 INFO:teuthology.orchestra.run.smithi027.stdout:service_name: grafana 2024-03-11T22:20:42.433 INFO:teuthology.orchestra.run.smithi027.stdout:placement: 2024-03-11T22:20:42.433 INFO:teuthology.orchestra.run.smithi027.stdout: count: 1 2024-03-11T22:20:42.433 INFO:teuthology.orchestra.run.smithi027.stdout: hosts: 2024-03-11T22:20:42.433 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi174=a 2024-03-11T22:20:42.433 INFO:teuthology.orchestra.run.smithi027.stdout:spec: 2024-03-11T22:20:42.433 INFO:teuthology.orchestra.run.smithi027.stdout: anonymous_access: true 2024-03-11T22:20:42.433 INFO:teuthology.orchestra.run.smithi027.stdout: protocol: https 2024-03-11T22:20:42.433 INFO:teuthology.orchestra.run.smithi027.stdout:status: 2024-03-11T22:20:42.433 INFO:teuthology.orchestra.run.smithi027.stdout: created: '2024-03-11T22:19:38.901279Z' 2024-03-11T22:20:42.433 INFO:teuthology.orchestra.run.smithi027.stdout: ports: 2024-03-11T22:20:42.433 INFO:teuthology.orchestra.run.smithi027.stdout: - 3000 2024-03-11T22:20:42.433 INFO:teuthology.orchestra.run.smithi027.stdout: running: 0 2024-03-11T22:20:42.433 INFO:teuthology.orchestra.run.smithi027.stdout: size: 1 2024-03-11T22:20:42.433 INFO:teuthology.orchestra.run.smithi027.stdout:--- 2024-03-11T22:20:42.434 INFO:teuthology.orchestra.run.smithi027.stdout:service_type: mgr 2024-03-11T22:20:42.434 INFO:teuthology.orchestra.run.smithi027.stdout:service_name: mgr 2024-03-11T22:20:42.434 INFO:teuthology.orchestra.run.smithi027.stdout:placement: 2024-03-11T22:20:42.434 INFO:teuthology.orchestra.run.smithi027.stdout: count: 3 2024-03-11T22:20:42.434 INFO:teuthology.orchestra.run.smithi027.stdout: hosts: 2024-03-11T22:20:42.434 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi027=y 2024-03-11T22:20:42.434 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi174=x 2024-03-11T22:20:42.434 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi183=z 2024-03-11T22:20:42.434 INFO:teuthology.orchestra.run.smithi027.stdout:status: 2024-03-11T22:20:42.434 INFO:teuthology.orchestra.run.smithi027.stdout: created: '2024-03-11T22:18:01.549543Z' 2024-03-11T22:20:42.434 INFO:teuthology.orchestra.run.smithi027.stdout: last_refresh: '2024-03-11T22:20:06.452830Z' 2024-03-11T22:20:42.434 INFO:teuthology.orchestra.run.smithi027.stdout: running: 3 2024-03-11T22:20:42.434 INFO:teuthology.orchestra.run.smithi027.stdout: size: 3 2024-03-11T22:20:42.434 INFO:teuthology.orchestra.run.smithi027.stdout:--- 2024-03-11T22:20:42.434 INFO:teuthology.orchestra.run.smithi027.stdout:service_type: mon 2024-03-11T22:20:42.435 INFO:teuthology.orchestra.run.smithi027.stdout:service_name: mon 2024-03-11T22:20:42.435 INFO:teuthology.orchestra.run.smithi027.stdout:placement: 2024-03-11T22:20:42.435 INFO:teuthology.orchestra.run.smithi027.stdout: count: 3 2024-03-11T22:20:42.435 INFO:teuthology.orchestra.run.smithi027.stdout: hosts: 2024-03-11T22:20:42.435 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi027:172.21.15.27=a 2024-03-11T22:20:42.435 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi174:172.21.15.174=b 2024-03-11T22:20:42.435 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi183:172.21.15.183=c 2024-03-11T22:20:42.435 INFO:teuthology.orchestra.run.smithi027.stdout:status: 2024-03-11T22:20:42.435 INFO:teuthology.orchestra.run.smithi027.stdout: created: '2024-03-11T22:17:35.815028Z' 2024-03-11T22:20:42.435 INFO:teuthology.orchestra.run.smithi027.stdout: last_refresh: '2024-03-11T22:20:06.452726Z' 2024-03-11T22:20:42.435 INFO:teuthology.orchestra.run.smithi027.stdout: running: 3 2024-03-11T22:20:42.435 INFO:teuthology.orchestra.run.smithi027.stdout: size: 3 2024-03-11T22:20:42.435 INFO:teuthology.orchestra.run.smithi027.stdout:--- 2024-03-11T22:20:42.435 INFO:teuthology.orchestra.run.smithi027.stdout:service_type: node-exporter 2024-03-11T22:20:42.435 INFO:teuthology.orchestra.run.smithi027.stdout:service_name: node-exporter 2024-03-11T22:20:42.435 INFO:teuthology.orchestra.run.smithi027.stdout:placement: 2024-03-11T22:20:42.436 INFO:teuthology.orchestra.run.smithi027.stdout: count: 3 2024-03-11T22:20:42.436 INFO:teuthology.orchestra.run.smithi027.stdout: hosts: 2024-03-11T22:20:42.436 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi027=a 2024-03-11T22:20:42.436 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi174=b 2024-03-11T22:20:42.436 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi183=c 2024-03-11T22:20:42.436 INFO:teuthology.orchestra.run.smithi027.stdout:status: 2024-03-11T22:20:42.436 INFO:teuthology.orchestra.run.smithi027.stdout: created: '2024-03-11T22:19:33.489088Z' 2024-03-11T22:20:42.436 INFO:teuthology.orchestra.run.smithi027.stdout: ports: 2024-03-11T22:20:42.436 INFO:teuthology.orchestra.run.smithi027.stdout: - 9100 2024-03-11T22:20:42.436 INFO:teuthology.orchestra.run.smithi027.stdout: running: 0 2024-03-11T22:20:42.436 INFO:teuthology.orchestra.run.smithi027.stdout: size: 3 2024-03-11T22:20:42.436 INFO:teuthology.orchestra.run.smithi027.stdout:--- 2024-03-11T22:20:42.436 INFO:teuthology.orchestra.run.smithi027.stdout:service_type: osd 2024-03-11T22:20:42.436 INFO:teuthology.orchestra.run.smithi027.stdout:service_name: osd 2024-03-11T22:20:42.436 INFO:teuthology.orchestra.run.smithi027.stdout:unmanaged: true 2024-03-11T22:20:42.437 INFO:teuthology.orchestra.run.smithi027.stdout:spec: 2024-03-11T22:20:42.437 INFO:teuthology.orchestra.run.smithi027.stdout: filter_logic: AND 2024-03-11T22:20:42.437 INFO:teuthology.orchestra.run.smithi027.stdout: objectstore: bluestore 2024-03-11T22:20:42.437 INFO:teuthology.orchestra.run.smithi027.stdout:status: 2024-03-11T22:20:42.437 INFO:teuthology.orchestra.run.smithi027.stdout: container_image_id: a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T22:20:42.437 INFO:teuthology.orchestra.run.smithi027.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:20:42.437 INFO:teuthology.orchestra.run.smithi027.stdout: last_refresh: '2024-03-11T22:20:06.452913Z' 2024-03-11T22:20:42.437 INFO:teuthology.orchestra.run.smithi027.stdout: running: 3 2024-03-11T22:20:42.437 INFO:teuthology.orchestra.run.smithi027.stdout: size: 3 2024-03-11T22:20:42.437 INFO:teuthology.orchestra.run.smithi027.stdout:--- 2024-03-11T22:20:42.437 INFO:teuthology.orchestra.run.smithi027.stdout:service_type: prometheus 2024-03-11T22:20:42.437 INFO:teuthology.orchestra.run.smithi027.stdout:service_name: prometheus 2024-03-11T22:20:42.437 INFO:teuthology.orchestra.run.smithi027.stdout:placement: 2024-03-11T22:20:42.437 INFO:teuthology.orchestra.run.smithi027.stdout: count: 1 2024-03-11T22:20:42.437 INFO:teuthology.orchestra.run.smithi027.stdout: hosts: 2024-03-11T22:20:42.437 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi174=a 2024-03-11T22:20:42.438 INFO:teuthology.orchestra.run.smithi027.stdout:status: 2024-03-11T22:20:42.438 INFO:teuthology.orchestra.run.smithi027.stdout: created: '2024-03-11T22:19:30.813157Z' 2024-03-11T22:20:42.438 INFO:teuthology.orchestra.run.smithi027.stdout: last_refresh: '2024-03-11T22:20:07.313259Z' 2024-03-11T22:20:42.438 INFO:teuthology.orchestra.run.smithi027.stdout: ports: 2024-03-11T22:20:42.438 INFO:teuthology.orchestra.run.smithi027.stdout: - 9095 2024-03-11T22:20:42.438 INFO:teuthology.orchestra.run.smithi027.stdout: running: 1 2024-03-11T22:20:42.438 INFO:teuthology.orchestra.run.smithi027.stdout: size: 1 2024-03-11T22:20:42.438 INFO:teuthology.orchestra.run.smithi027.stdout:--- 2024-03-11T22:20:42.438 INFO:teuthology.orchestra.run.smithi027.stdout:service_type: rgw 2024-03-11T22:20:42.438 INFO:teuthology.orchestra.run.smithi027.stdout:service_id: foo.a 2024-03-11T22:20:42.438 INFO:teuthology.orchestra.run.smithi027.stdout:service_name: rgw.foo.a 2024-03-11T22:20:42.438 INFO:teuthology.orchestra.run.smithi027.stdout:placement: 2024-03-11T22:20:42.438 INFO:teuthology.orchestra.run.smithi027.stdout: count: 1 2024-03-11T22:20:42.438 INFO:teuthology.orchestra.run.smithi027.stdout: hosts: 2024-03-11T22:20:42.438 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi027=foo.a 2024-03-11T22:20:42.439 INFO:teuthology.orchestra.run.smithi027.stdout:status: 2024-03-11T22:20:42.439 INFO:teuthology.orchestra.run.smithi027.stdout: created: '2024-03-11T22:19:31.788367Z' 2024-03-11T22:20:42.439 INFO:teuthology.orchestra.run.smithi027.stdout: last_refresh: '2024-03-11T22:20:10.453666Z' 2024-03-11T22:20:42.439 INFO:teuthology.orchestra.run.smithi027.stdout: ports: 2024-03-11T22:20:42.439 INFO:teuthology.orchestra.run.smithi027.stdout: - 80 2024-03-11T22:20:42.439 INFO:teuthology.orchestra.run.smithi027.stdout: running: 1 2024-03-11T22:20:42.439 INFO:teuthology.orchestra.run.smithi027.stdout: size: 1 2024-03-11T22:20:42.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:42 smithi174 ceph-mon[30085]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:42.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:42 smithi183 ceph-mon[30906]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:42.754 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:42 smithi027 ceph-mon[25797]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:43.116 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-11T22:20:43.366 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:44.341 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:44 smithi174 ceph-mon[30085]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-11T22:20:44.341 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:44 smithi174 ceph-mon[30085]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:44.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:44 smithi027 ceph-mon[25797]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-11T22:20:44.479 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:44 smithi027 ceph-mon[25797]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:44.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:44 smithi183 ceph-mon[30906]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-11T22:20:44.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:44 smithi183 ceph-mon[30906]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:44.735 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:44 smithi174 systemd[1]: Starting Ceph grafana.a for c71f9f14-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:20:45.005 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:44 smithi174 podman[42363]: 2024-03-11 22:20:44.635927079 +0000 UTC m=+0.016864511 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-11T22:20:45.005 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:44 smithi174 podman[42363]: 2024-03-11 22:20:44.743521207 +0000 UTC m=+0.124458639 container create 1e6f31f66d90bc6ae103597f0368dca3597b2e34655b8423680a34f918b0bfc2 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., description=Ceph Grafana Container, release=239.1651231664, com.redhat.component=ubi8-container, build-date=2022-04-29T11:48:11.055897, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , distribution-scope=public, name=ubi8, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-11T22:20:45.005 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:44 smithi174 podman[42363]: 2024-03-11 22:20:44.971008444 +0000 UTC m=+0.351945897 container init 1e6f31f66d90bc6ae103597f0368dca3597b2e34655b8423680a34f918b0bfc2 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, name=ubi8, release=239.1651231664, vcs-type=git, description=Ceph Grafana Container, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., io.openshift.expose-services=, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , version=8.5, io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-03-11T22:20:45.005 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:44 smithi174 podman[42363]: 2024-03-11 22:20:44.978061969 +0000 UTC m=+0.358999401 container start 1e6f31f66d90bc6ae103597f0368dca3597b2e34655b8423680a34f918b0bfc2 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a, vendor=Red Hat, Inc., architecture=x86_64, maintainer=Paul Cuzner , io.openshift.expose-services=, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-04-29T11:48:11.055897, version=8.5, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, description=Ceph Grafana Container, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, release=239.1651231664) 2024-03-11T22:20:45.288 INFO:teuthology.orchestra.run.smithi027.stdout:osd 3 38s ago - 2024-03-11T22:20:45.324 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 bash[42363]: 1e6f31f66d90bc6ae103597f0368dca3597b2e34655b8423680a34f918b0bfc2 2024-03-11T22:20:45.324 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 systemd[1]: Started Ceph grafana.a for c71f9f14-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:20:45.324 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=settings t=2024-03-11T22:20:45.129982132Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-11T22:20:45.324 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=settings t=2024-03-11T22:20:45.130197895Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-11T22:20:45.324 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=settings t=2024-03-11T22:20:45.130216091Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-11T22:20:45.324 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=settings t=2024-03-11T22:20:45.130223461Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-11T22:20:45.324 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=settings t=2024-03-11T22:20:45.13022954Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-11T22:20:45.324 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=settings t=2024-03-11T22:20:45.130235864Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-11T22:20:45.325 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=settings t=2024-03-11T22:20:45.130241736Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-11T22:20:45.325 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=settings t=2024-03-11T22:20:45.13024756Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-11T22:20:45.325 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=settings t=2024-03-11T22:20:45.130253441Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-11T22:20:45.325 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=settings t=2024-03-11T22:20:45.130259207Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-11T22:20:45.325 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=settings t=2024-03-11T22:20:45.130264781Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-11T22:20:45.325 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=settings t=2024-03-11T22:20:45.130270715Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-11T22:20:45.325 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=settings t=2024-03-11T22:20:45.130276508Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-11T22:20:45.325 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=settings t=2024-03-11T22:20:45.130282365Z level=info msg="App mode production" 2024-03-11T22:20:45.325 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=sqlstore t=2024-03-11T22:20:45.130325285Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-11T22:20:45.325 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=sqlstore t=2024-03-11T22:20:45.130341097Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-11T22:20:45.325 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.131240007Z level=info msg="Starting DB migrations" 2024-03-11T22:20:45.325 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.131314336Z level=info msg="Executing migration" id="create migration_log table" 2024-03-11T22:20:45.326 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.131840256Z level=info msg="Executing migration" id="create user table" 2024-03-11T22:20:45.326 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.132304165Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-11T22:20:45.326 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.132785687Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-11T22:20:45.326 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.133228501Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-11T22:20:45.326 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.133775189Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-11T22:20:45.327 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.134193411Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-11T22:20:45.327 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.134907475Z level=info msg="Executing migration" id="create user table v2" 2024-03-11T22:20:45.327 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.135310298Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-11T22:20:45.327 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.13569148Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-11T22:20:45.327 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.136187996Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-11T22:20:45.327 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.136506972Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-11T22:20:45.327 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.136871825Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-11T22:20:45.327 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.137284784Z level=info msg="Executing migration" id="Update user table charset" 2024-03-11T22:20:45.327 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.13754971Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-11T22:20:45.327 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.137968347Z level=info msg="Executing migration" id="Add missing user data" 2024-03-11T22:20:45.327 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.138318738Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-11T22:20:45.327 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.138698304Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-11T22:20:45.327 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.139113731Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-11T22:20:45.327 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.139497284Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-11T22:20:45.328 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.140633805Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-11T22:20:45.328 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.141079491Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-11T22:20:45.328 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.14152625Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-11T22:20:45.328 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.142280056Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-11T22:20:45.328 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.142729261Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-11T22:20:45.328 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.143507851Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-11T22:20:45.328 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.144340612Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-11T22:20:45.328 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.144946461Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-11T22:20:45.328 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.145359707Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-11T22:20:45.328 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.146135268Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-11T22:20:45.328 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.146615828Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-11T22:20:45.328 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.147300814Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-11T22:20:45.328 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.14814932Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-11T22:20:45.328 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.148672048Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-11T22:20:45.329 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.149102722Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-11T22:20:45.329 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.149860141Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-11T22:20:45.329 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.150292181Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-11T22:20:45.329 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.15067249Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-11T22:20:45.329 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.151548176Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-11T22:20:45.329 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.15258832Z level=info msg="Executing migration" id="create star table" 2024-03-11T22:20:45.329 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.153287555Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-11T22:20:45.329 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.154166178Z level=info msg="Executing migration" id="create org table v1" 2024-03-11T22:20:45.329 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.15492867Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-11T22:20:45.329 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.155744993Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-11T22:20:45.329 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.15659826Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-11T22:20:45.329 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.157475715Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-11T22:20:45.329 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.157889715Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-11T22:20:45.329 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.158352328Z level=info msg="Executing migration" id="Update org table charset" 2024-03-11T22:20:45.329 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.159214388Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-11T22:20:45.330 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.159740532Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-11T22:20:45.330 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.160075882Z level=info msg="Executing migration" id="create dashboard table" 2024-03-11T22:20:45.330 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.160937159Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-11T22:20:45.330 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.161945681Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-11T22:20:45.330 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.162731222Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-11T22:20:45.330 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.163628651Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-11T22:20:45.330 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.166289772Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-11T22:20:45.330 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.16899253Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-11T22:20:45.330 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.17180889Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-11T22:20:45.330 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.174629289Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-11T22:20:45.330 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.177406669Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-11T22:20:45.330 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.179847163Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-11T22:20:45.330 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.182811325Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-11T22:20:45.330 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.185524259Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-11T22:20:45.330 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.18818526Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-11T22:20:45.331 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.191040997Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-11T22:20:45.331 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.193812828Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-11T22:20:45.331 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.196582283Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-11T22:20:45.331 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.199080773Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-11T22:20:45.331 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.201954269Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-11T22:20:45.331 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.204768157Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-11T22:20:45.331 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.207403643Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-11T22:20:45.331 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.210090708Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-11T22:20:45.331 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.212852001Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-11T22:20:45.331 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.215158949Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-11T22:20:45.331 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.215680616Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-11T22:20:45.331 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.218385024Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-11T22:20:45.331 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.221163918Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-11T22:20:45.331 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.223807058Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-11T22:20:45.332 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.226415667Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-11T22:20:45.332 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.229014546Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-11T22:20:45.332 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.23177032Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-11T22:20:45.332 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.234443561Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-11T22:20:45.332 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.237172523Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-11T22:20:45.332 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.239725739Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-11T22:20:45.332 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.242232205Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-11T22:20:45.333 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.244869333Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-11T22:20:45.333 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.247652614Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-11T22:20:45.333 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.250171643Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-11T22:20:45.333 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.252950741Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-11T22:20:45.333 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.255563075Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-11T22:20:45.333 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.258195072Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-11T22:20:45.333 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.258866814Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-11T22:20:45.333 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.259500043Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-11T22:20:45.333 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.260299639Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-11T22:20:45.333 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.261327277Z level=info msg="Executing migration" id="create data_source table" 2024-03-11T22:20:45.333 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.261944999Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-11T22:20:45.333 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.262826059Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-11T22:20:45.333 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.263557869Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-11T22:20:45.333 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.264477732Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-11T22:20:45.333 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.265430685Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-11T22:20:45.333 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.266485954Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-11T22:20:45.334 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.267378654Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-11T22:20:45.334 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.268222854Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-11T22:20:45.334 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.269098536Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-11T22:20:45.334 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.269895922Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-11T22:20:45.334 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.270781472Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-11T22:20:45.334 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.271619847Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-11T22:20:45.334 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.272486051Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-11T22:20:45.334 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.273429259Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-11T22:20:45.334 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.274277589Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-11T22:20:45.334 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.275024022Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-11T22:20:45.334 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.275884334Z level=info msg="Executing migration" id="Add uid column" 2024-03-11T22:20:45.334 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.276761146Z level=info msg="Executing migration" id="Update uid value" 2024-03-11T22:20:45.334 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.277726788Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-11T22:20:45.334 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.278407957Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-11T22:20:45.334 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.281382361Z level=info msg="Executing migration" id="create api_key table" 2024-03-11T22:20:45.335 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.286725504Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-11T22:20:45.335 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.287179919Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-11T22:20:45.335 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.289892066Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-11T22:20:45.335 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.29242039Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-11T22:20:45.335 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.295210197Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-11T22:20:45.335 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.297933841Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-11T22:20:45.335 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.300748596Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-11T22:20:45.335 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.303443471Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-11T22:20:45.335 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.30604138Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-11T22:20:45.335 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.308900748Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-11T22:20:45.335 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.31147895Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-11T22:20:45.335 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.31427Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-11T22:20:45.335 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.316924961Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-11T22:20:45.335 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.319740591Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-11T22:20:45.335 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.322386124Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-11T22:20:45.335 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.325083646Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-11T22:20:45.579 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.3277642Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-11T22:20:45.579 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.330264873Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-11T22:20:45.579 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.332760221Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-11T22:20:45.580 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.33543271Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-11T22:20:45.580 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.338374164Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-11T22:20:45.580 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.340913237Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-11T22:20:45.580 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.343580331Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-11T22:20:45.580 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.346279831Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-11T22:20:45.580 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.3489827Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-11T22:20:45.580 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.351624062Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-11T22:20:45.580 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.354356046Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-11T22:20:45.580 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.357035887Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-11T22:20:45.580 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.359836874Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-11T22:20:45.580 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.362497783Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-11T22:20:45.581 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.364965052Z level=info msg="Executing migration" id="create quota table v1" 2024-03-11T22:20:45.581 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.367634094Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-11T22:20:45.581 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.370365521Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-11T22:20:45.581 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.372799939Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-11T22:20:45.581 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.373783422Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-11T22:20:45.581 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.374680043Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-11T22:20:45.581 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.375382234Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-11T22:20:45.581 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.376149147Z level=info msg="Executing migration" id="create session table" 2024-03-11T22:20:45.581 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.376652226Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-11T22:20:45.581 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.37698235Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-11T22:20:45.581 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.377847452Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-11T22:20:45.582 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.378732551Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-11T22:20:45.582 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.379414442Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-11T22:20:45.582 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.380156776Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-11T22:20:45.582 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.380461197Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-11T22:20:45.582 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.381070043Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-11T22:20:45.582 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.381502889Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-11T22:20:45.582 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.381896344Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-11T22:20:45.582 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.382832405Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-11T22:20:45.582 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.383623913Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-11T22:20:45.582 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.384552721Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-11T22:20:45.582 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.385319873Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-11T22:20:45.582 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.38632926Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-11T22:20:45.582 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.387058861Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-11T22:20:45.583 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.387922128Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-11T22:20:45.583 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.388998588Z level=info msg="Executing migration" id="create alert table v1" 2024-03-11T22:20:45.583 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.38967652Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-11T22:20:45.583 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.390461253Z level=info msg="Executing migration" id="add index alert state" 2024-03-11T22:20:45.583 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.391459835Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-11T22:20:45.583 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.392170569Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-11T22:20:45.583 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.393026908Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-11T22:20:45.583 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.393897175Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-11T22:20:45.583 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.394844081Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-11T22:20:45.583 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.39635246Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-11T22:20:45.583 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.397214987Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-11T22:20:45.583 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.398068352Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-11T22:20:45.584 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.398900955Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-11T22:20:45.584 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.399762169Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-11T22:20:45.584 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.400740409Z level=info msg="Executing migration" id="Add column is_default" 2024-03-11T22:20:45.584 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.403612086Z level=info msg="Executing migration" id="Add column frequency" 2024-03-11T22:20:45.584 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.406114215Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-11T22:20:45.584 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.40896254Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-11T22:20:45.584 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.41165087Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-11T22:20:45.584 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.414157138Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-11T22:20:45.584 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.417101931Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-11T22:20:45.584 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.41972804Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-11T22:20:45.584 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.422461806Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-11T22:20:45.585 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.425149992Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-11T22:20:45.585 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.427877935Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-11T22:20:45.585 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.430400519Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-11T22:20:45.585 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.43312902Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-11T22:20:45.585 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.435863953Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-11T22:20:45.585 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.438473711Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-11T22:20:45.585 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.441312311Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-11T22:20:45.585 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.443979337Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-11T22:20:45.585 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.446824007Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-11T22:20:45.585 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.449656391Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-11T22:20:45.585 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.45240399Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-11T22:20:45.586 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.45480919Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-11T22:20:45.586 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.457501038Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-11T22:20:45.586 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.460151968Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-11T22:20:45.586 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.462894922Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-11T22:20:45.586 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.465521661Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-11T22:20:45.586 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.468233099Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-11T22:20:45.586 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.470765315Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-11T22:20:45.586 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.4734672Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-11T22:20:45.586 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.47624787Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-11T22:20:45.586 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.478923357Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-11T22:20:45.587 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.481570366Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-11T22:20:45.587 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.48408244Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-11T22:20:45.587 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.486797916Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-11T22:20:45.587 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.489391896Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-11T22:20:45.587 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.491984083Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-11T22:20:45.587 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.494755548Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-11T22:20:45.587 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.496508123Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-11T22:20:45.587 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.499159386Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-11T22:20:45.587 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.501808133Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-11T22:20:45.587 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.504617556Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-11T22:20:45.587 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.507372097Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-11T22:20:45.587 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.510123824Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-11T22:20:45.588 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.512726092Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-11T22:20:45.588 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.515327034Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-11T22:20:45.588 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.517920776Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-11T22:20:45.588 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.520673933Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-11T22:20:45.588 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.523400638Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-11T22:20:45.588 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.525979254Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-11T22:20:45.588 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.528784315Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-11T22:20:45.588 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.531402829Z level=info msg="Executing migration" id="Move region to single row" 2024-03-11T22:20:45.588 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.53208854Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-11T22:20:45.588 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.532855735Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-11T22:20:45.588 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.533641974Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-11T22:20:45.588 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.534453677Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-11T22:20:45.588 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.535240561Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-11T22:20:45.588 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.536058924Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-11T22:20:45.588 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.53686054Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-11T22:20:45.589 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.537716568Z level=info msg="Executing migration" id="create test_data table" 2024-03-11T22:20:45.589 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.538438528Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-11T22:20:45.589 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.538856733Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-11T22:20:45.589 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.539665953Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-11T22:20:45.589 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.540477791Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-11T22:20:45.589 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.541217555Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-11T22:20:45.589 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.541577288Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-11T22:20:45.589 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.542097873Z level=info msg="Executing migration" id="create team table" 2024-03-11T22:20:45.589 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.542879557Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-11T22:20:45.589 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.543664736Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-11T22:20:45.589 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.544617537Z level=info msg="Executing migration" id="create team member table" 2024-03-11T22:20:45.589 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.545300528Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-11T22:20:45.589 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.546251084Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-11T22:20:45.589 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.546941223Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-11T22:20:45.590 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.547818343Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-11T22:20:45.590 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.548680648Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-11T22:20:45.590 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.549505422Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-11T22:20:45.590 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.550298229Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-11T22:20:45.590 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.551102133Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-11T22:20:45.590 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.551866993Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-11T22:20:45.590 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.552746228Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-11T22:20:45.590 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.553204447Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-11T22:20:45.590 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.55407031Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-11T22:20:45.590 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.554904508Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-11T22:20:45.591 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.555346814Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-11T22:20:45.591 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.55577963Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-11T22:20:45.591 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.556180943Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-11T22:20:45.591 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.556474514Z level=info msg="Executing migration" id="create tag table" 2024-03-11T22:20:45.591 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.556860385Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-11T22:20:45.591 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.557302621Z level=info msg="Executing migration" id="create login attempt table" 2024-03-11T22:20:45.591 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.557740554Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-11T22:20:45.591 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.558189861Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-11T22:20:45.591 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.558592556Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-11T22:20:45.592 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.560526182Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-11T22:20:45.592 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.560931549Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-11T22:20:45.592 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.561382173Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-11T22:20:45.592 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.561688998Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-11T22:20:45.592 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.562063408Z level=info msg="Executing migration" id="create user auth table" 2024-03-11T22:20:45.592 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.56241515Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-11T22:20:45.592 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.562847584Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-11T22:20:45.592 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.563164921Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-11T22:20:45.592 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.56390183Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-11T22:20:45.592 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.564632715Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-11T22:20:45.592 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.565366916Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-11T22:20:45.592 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.566146947Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-11T22:20:45.592 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.566577523Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-11T22:20:45.592 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.56734659Z level=info msg="Executing migration" id="create server_lock table" 2024-03-11T22:20:45.592 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.567757282Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-11T22:20:45.593 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.568218724Z level=info msg="Executing migration" id="create user auth token table" 2024-03-11T22:20:45.593 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.568651656Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-11T22:20:45.593 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.569086842Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-11T22:20:45.593 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.56952585Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-11T22:20:45.593 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.57000485Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-11T22:20:45.593 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.570772939Z level=info msg="Executing migration" id="create cache_data table" 2024-03-11T22:20:45.593 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.571180717Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-11T22:20:45.593 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.571623073Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-11T22:20:45.593 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.572087816Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-11T22:20:45.593 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.572896771Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-11T22:20:45.593 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.573216053Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-11T22:20:45.593 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.573502779Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-11T22:20:45.593 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.573972022Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-11T22:20:45.593 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.574826421Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-11T22:20:45.593 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.575611051Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-11T22:20:45.594 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.575935324Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-11T22:20:45.594 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.576419343Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-11T22:20:45.594 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.577262723Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-11T22:20:45.826 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.580042146Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-11T22:20:45.826 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.582733038Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-11T22:20:45.827 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.583547198Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-11T22:20:45.827 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.586346802Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-11T22:20:45.827 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.58916078Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-11T22:20:45.827 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.591688359Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-11T22:20:45.827 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.592151024Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-11T22:20:45.827 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.594834141Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-11T22:20:45.827 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.595154165Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-11T22:20:45.827 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.597665209Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-11T22:20:45.827 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.598112715Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-11T22:20:45.828 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.600487274Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-11T22:20:45.828 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.60111645Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-11T22:20:45.828 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.601960617Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-11T22:20:45.828 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.604894004Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-11T22:20:45.828 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.607443269Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-11T22:20:45.828 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.61083741Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-11T22:20:45.829 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.614181383Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-11T22:20:45.829 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.616882647Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-11T22:20:45.829 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.619552792Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-11T22:20:45.829 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.620381874Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-11T22:20:45.829 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.622206532Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-11T22:20:45.829 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.62498892Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-11T22:20:45.829 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.627577348Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-11T22:20:45.829 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.630193284Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-11T22:20:45.830 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.630508553Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-11T22:20:45.830 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.633137565Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-11T22:20:45.830 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.636024921Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-11T22:20:45.830 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.638417424Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-11T22:20:45.830 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.638932557Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-11T22:20:45.830 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.641740059Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-11T22:20:45.830 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.644432981Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-11T22:20:45.830 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.646984556Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-11T22:20:45.830 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.647487154Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-11T22:20:45.830 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.650243408Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-11T22:20:45.830 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.652762638Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-11T22:20:45.831 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.65368246Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-11T22:20:45.831 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.654201049Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-11T22:20:45.831 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.655105277Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-11T22:20:45.831 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.655754588Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-11T22:20:45.831 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.656060967Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-11T22:20:45.831 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.656950009Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-11T22:20:45.831 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.6578297Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-11T22:20:45.831 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.658658876Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-11T22:20:45.831 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.659506471Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-11T22:20:45.831 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.66035902Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-11T22:20:45.831 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.661185887Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-11T22:20:45.832 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.662021008Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-11T22:20:45.832 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.662893047Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-11T22:20:45.832 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.663206692Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-11T22:20:45.832 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.665818702Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-11T22:20:45.832 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.668555122Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-11T22:20:45.832 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.671109575Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-11T22:20:45.832 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.671589779Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-11T22:20:45.832 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.672471534Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-11T22:20:45.832 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.673386819Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-11T22:20:45.832 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.674209311Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-11T22:20:45.832 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.674680355Z level=info msg="Executing migration" id="create alert_image table" 2024-03-11T22:20:45.833 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.675625634Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-11T22:20:45.833 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.676290711Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-11T22:20:45.833 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.677145769Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-11T22:20:45.833 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.677816609Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-11T22:20:45.833 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.678238389Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T22:20:45.833 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.678329469Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T22:20:45.833 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.679102902Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-11T22:20:45.833 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.679686074Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-11T22:20:45.833 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.680148409Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-11T22:20:45.833 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.680281847Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-11T22:20:45.833 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.680391139Z level=info msg="alerts found to migrate" alerts=0 2024-03-11T22:20:45.833 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.68094363Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-11T22:20:45.833 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.681442537Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-11T22:20:45.834 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.681923455Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-11T22:20:45.834 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.682699835Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-11T22:20:45.834 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.683482049Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-11T22:20:45.834 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.684481395Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-11T22:20:45.834 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.684827474Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-11T22:20:45.834 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.685679623Z level=info msg="Executing migration" id="create data_keys table" 2024-03-11T22:20:45.834 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.686654905Z level=info msg="Executing migration" id="create secrets table" 2024-03-11T22:20:45.834 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.687380257Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-11T22:20:45.834 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.69169001Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-11T22:20:45.834 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.692597533Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-11T22:20:45.834 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.693427129Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-11T22:20:45.834 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.697540515Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-11T22:20:45.835 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.701750353Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-11T22:20:45.835 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.70257603Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-11T22:20:45.835 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.703577696Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-11T22:20:45.835 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.704285919Z level=info msg="Executing migration" id="create permission table" 2024-03-11T22:20:45.835 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.704697238Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-11T22:20:45.835 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.70555816Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-11T22:20:45.835 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.706385631Z level=info msg="Executing migration" id="create role table" 2024-03-11T22:20:45.835 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.707172388Z level=info msg="Executing migration" id="add column display_name" 2024-03-11T22:20:45.836 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.708113346Z level=info msg="Executing migration" id="add column group_name" 2024-03-11T22:20:45.836 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.710822773Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-11T22:20:45.836 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.713609389Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-11T22:20:45.836 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.716243473Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-11T22:20:45.836 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.716693832Z level=info msg="Executing migration" id="create team role table" 2024-03-11T22:20:45.837 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.719576765Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-11T22:20:45.837 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.722232095Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-11T22:20:45.837 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.724847673Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-11T22:20:45.837 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.727620611Z level=info msg="Executing migration" id="create user role table" 2024-03-11T22:20:45.837 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.73037496Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-11T22:20:45.837 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.732956828Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-11T22:20:45.837 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.735899075Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-11T22:20:45.837 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.736402214Z level=info msg="Executing migration" id="create builtin role table" 2024-03-11T22:20:45.837 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.739180294Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-11T22:20:45.837 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.741997238Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-11T22:20:45.837 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.744618008Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-11T22:20:45.837 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.745634007Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-11T22:20:45.838 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.748509227Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-11T22:20:45.838 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.751151629Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-11T22:20:45.838 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.753804744Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-11T22:20:45.838 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.754273464Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-11T22:20:45.838 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.757189934Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-11T22:20:45.838 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.759864998Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-11T22:20:45.838 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.762360938Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-11T22:20:45.838 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.76296234Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-11T22:20:45.838 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.765693972Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-11T22:20:45.838 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.768208365Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-11T22:20:45.839 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.770989324Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-11T22:20:45.839 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.771570128Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-11T22:20:45.839 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.77444756Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-11T22:20:45.839 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.777052079Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-11T22:20:45.839 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.779624841Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-11T22:20:45.839 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.780238174Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-11T22:20:45.839 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.782930335Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-11T22:20:45.839 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.785540288Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-11T22:20:45.839 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.786428006Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-11T22:20:45.839 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.786869363Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-11T22:20:45.839 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.787710302Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-11T22:20:45.839 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.788521371Z level=info msg="Executing migration" id="add correlation config column" 2024-03-11T22:20:45.840 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.78953308Z level=info msg="Executing migration" id="create entity_events table" 2024-03-11T22:20:45.840 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.790061796Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-11T22:20:45.840 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.790803035Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T22:20:45.840 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.790886539Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T22:20:45.840 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.791579012Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T22:20:45.840 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.791657766Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T22:20:45.840 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.792369197Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-11T22:20:45.840 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.793256738Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-11T22:20:45.840 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.794004442Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-11T22:20:45.840 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.794944108Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T22:20:45.840 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.795577663Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-11T22:20:45.841 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.796000451Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T22:20:45.841 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.796836083Z level=info msg="Executing migration" id="Drop public config table" 2024-03-11T22:20:45.841 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.797599042Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-11T22:20:45.841 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.798570951Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-11T22:20:45.841 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.799130066Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T22:20:45.841 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.800067793Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-11T22:20:45.841 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.800732653Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-11T22:20:45.841 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.803728295Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-11T22:20:45.841 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.804722483Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-11T22:20:45.841 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.805715274Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-11T22:20:45.841 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.806677491Z level=info msg="Executing migration" id="add share column" 2024-03-11T22:20:45.841 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.80785381Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-11T22:20:45.841 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.808870501Z level=info msg="Executing migration" id="create file table" 2024-03-11T22:20:45.842 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.809511827Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-11T22:20:45.842 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.810370701Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-11T22:20:45.842 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.811176646Z level=info msg="Executing migration" id="create file_meta table" 2024-03-11T22:20:45.842 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.812078386Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-11T22:20:45.842 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.812514284Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-11T22:20:45.842 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.81332455Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-11T22:20:45.842 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.813728222Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-11T22:20:45.842 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.814196145Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-11T22:20:45.842 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.815044984Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-11T22:20:45.842 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.816079533Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-11T22:20:45.842 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.816388777Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-11T22:20:45.842 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.819299196Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-11T22:20:45.843 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.821892241Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-11T22:20:45.843 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.824655291Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-11T22:20:45.843 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.825095063Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-11T22:20:45.843 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.827699588Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-11T22:20:45.843 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.830440565Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-11T22:20:46.015 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-11T22:20:46.024 INFO:tasks.cephadm:Teardown begin 2024-03-11T22:20:46.024 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:20:46.058 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:20:46.084 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.83310512Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-11T22:20:46.084 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.843201459Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-11T22:20:46.084 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.845850673Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-11T22:20:46.084 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.848742137Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-11T22:20:46.085 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.852243204Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-11T22:20:46.085 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.852574522Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-11T22:20:46.085 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.855500466Z level=info msg="Executing migration" id="create folder table" 2024-03-11T22:20:46.085 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.858099571Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-11T22:20:46.085 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.860714929Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-11T22:20:46.085 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.861270312Z level=info msg="Executing migration" id="Update folder title length" 2024-03-11T22:20:46.085 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.864039908Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-11T22:20:46.085 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=migrator t=2024-03-11T22:20:45.86450032Z level=info msg="migrations completed" performed=485 skipped=0 duration=733.207892ms 2024-03-11T22:20:46.085 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=sqlstore t=2024-03-11T22:20:45.8648846Z level=info msg="Created default organization" 2024-03-11T22:20:46.085 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=plugin.loader t=2024-03-11T22:20:45.921855677Z level=info msg="Plugin registered" pluginID=input 2024-03-11T22:20:46.085 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=plugin.loader t=2024-03-11T22:20:45.948239072Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-11T22:20:46.085 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=plugin.loader t=2024-03-11T22:20:45.948256978Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-11T22:20:46.085 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=secrets t=2024-03-11T22:20:45.948289439Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-11T22:20:46.085 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=query_data t=2024-03-11T22:20:45.950367133Z level=info msg="Query Service initialization" 2024-03-11T22:20:46.085 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:45 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=live.push_http t=2024-03-11T22:20:45.952995274Z level=info msg="Live Push Gateway initialization" 2024-03-11T22:20:46.088 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:20:46.126 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-11T22:20:46.126 DEBUG:teuthology.orchestra.run.smithi027:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:20:46.141 DEBUG:teuthology.orchestra.run.smithi174:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:20:46.155 DEBUG:teuthology.orchestra.run.smithi183:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:20:46.182 INFO:tasks.cephadm:Stopping all daemons... 2024-03-11T22:20:46.182 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-11T22:20:46.182 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mon.a 2024-03-11T22:20:46.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:46 smithi027 ceph-mon[25797]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:46.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:46 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:46.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:46 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:46.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:46 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:46.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:46 smithi027 ceph-mon[25797]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:46.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:46 smithi027 ceph-mon[25797]: Deploying daemon node-exporter.a on smithi027 2024-03-11T22:20:46.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:46 smithi027 ceph-mon[25797]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:46.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:46 smithi174 ceph-mon[30085]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:46.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:46 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:46.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:46 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:46.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:46 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:46.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:46 smithi174 ceph-mon[30085]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:46.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:46 smithi174 ceph-mon[30085]: Deploying daemon node-exporter.a on smithi027 2024-03-11T22:20:46.505 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:46 smithi174 ceph-mon[30085]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:46.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:46 smithi183 ceph-mon[30906]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:46.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:46 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:46.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:46 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:46.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:46 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:46.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:46 smithi183 ceph-mon[30906]: from='mgr.14439 172.21.15.27:0/2700963176' entity='mgr.y' 2024-03-11T22:20:46.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:46 smithi183 ceph-mon[30906]: Deploying daemon node-exporter.a on smithi027 2024-03-11T22:20:46.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:46 smithi183 ceph-mon[30906]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:46.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:46 smithi027 systemd[1]: Stopping Ceph mon.a for c71f9f14-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:20:47.005 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:46 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=infra.usagestats.collector t=2024-03-11T22:20:46.558181827Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-11T22:20:47.005 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:46 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=provisioning.datasources t=2024-03-11T22:20:46.559389895Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-11T22:20:47.005 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:46 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=provisioning.datasources t=2024-03-11T22:20:46.570472423Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-11T22:20:47.006 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:46 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=provisioning.alerting t=2024-03-11T22:20:46.58137345Z level=info msg="starting to provision alerting" 2024-03-11T22:20:47.006 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:46 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=provisioning.alerting t=2024-03-11T22:20:46.581390874Z level=info msg="finished to provision alerting" 2024-03-11T22:20:47.006 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:46 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=ngalert.state.manager t=2024-03-11T22:20:46.584793411Z level=info msg="Warming state cache for startup" 2024-03-11T22:20:47.006 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:46 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=ngalert.state.manager t=2024-03-11T22:20:46.585043356Z level=info msg="State cache has been initialized" states=0 duration=248.901µs 2024-03-11T22:20:47.006 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:46 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=ngalert.multiorg.alertmanager t=2024-03-11T22:20:46.585177663Z level=info msg="starting MultiOrg Alertmanager" 2024-03-11T22:20:47.006 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:46 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=ticker t=2024-03-11T22:20:46.585278874Z level=info msg=starting first_tick=2024-03-11T22:20:50Z 2024-03-11T22:20:47.006 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:46 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=grafanaStorageLogger t=2024-03-11T22:20:46.585469293Z level=info msg="storage starting" 2024-03-11T22:20:47.006 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:46 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=http.server t=2024-03-11T22:20:46.58850793Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-11T22:20:47.006 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:20:46 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a[42377]: logger=sqlstore.transactions t=2024-03-11T22:20:46.606011026Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T22:20:47.022 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:46 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: [11/Mar/2024:22:20:46] ENGINE Bus STOPPING 2024-03-11T22:20:47.023 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:46 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: [11/Mar/2024:22:20:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T22:20:47.023 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:46 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: [11/Mar/2024:22:20:46] ENGINE Bus STOPPED 2024-03-11T22:20:47.023 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:46 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: [11/Mar/2024:22:20:46] ENGINE Bus STARTING 2024-03-11T22:20:47.023 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:46 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: [11/Mar/2024:22:20:46] ENGINE Serving on http://:::9283 2024-03-11T22:20:47.023 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:46 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: [11/Mar/2024:22:20:46] ENGINE Bus STARTED 2024-03-11T22:20:47.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:46 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-a[25774]: 2024-03-11T22:20:46.667+0000 7f27c7d52700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:20:47.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:46 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-a[25774]: 2024-03-11T22:20:46.667+0000 7f27c7d52700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-11T22:20:47.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:46 smithi027 podman[60068]: 2024-03-11 22:20:46.80101142 +0000 UTC m=+0.251906912 container died 77695fc9e008d15a63bfed6f337637aaeabadde59a3dd7e6ff22bcd268ed65ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-11T22:20:47.023 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:46 smithi027 bash[60151]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-11T22:20:47.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:47 smithi027 podman[60068]: 2024-03-11 22:20:47.215082276 +0000 UTC m=+0.665977767 container cleanup 77695fc9e008d15a63bfed6f337637aaeabadde59a3dd7e6ff22bcd268ed65ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, ceph=True) 2024-03-11T22:20:47.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:47 smithi027 bash[60068]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-a 2024-03-11T22:20:47.523 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:47 smithi027 bash[60151]: Getting image source signatures 2024-03-11T22:20:47.523 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:47 smithi027 bash[60151]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-11T22:20:47.523 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:47 smithi027 bash[60151]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-11T22:20:47.523 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:47 smithi027 bash[60151]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-11T22:20:47.924 INFO:journalctl@ceph.mon.a.smithi027.stdout:Mar 11 22:20:47 smithi027 podman[60164]: 2024-03-11 22:20:47.523339792 +0000 UTC m=+0.726965368 container remove 77695fc9e008d15a63bfed6f337637aaeabadde59a3dd7e6ff22bcd268ed65ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:20:47.938 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mon.a.service' 2024-03-11T22:20:48.014 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:20:48.014 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-11T22:20:48.014 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-11T22:20:48.014 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mon.b 2024-03-11T22:20:48.319 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:48 smithi174 systemd[1]: Stopping Ceph mon.b for c71f9f14-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:20:48.709 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:48 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-b[30062]: 2024-03-11T22:20:48.317+0000 7f4a6c585700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:20:48.709 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:48 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-b[30062]: 2024-03-11T22:20:48.317+0000 7f4a6c585700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-11T22:20:48.709 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:48 smithi174 podman[42498]: 2024-03-11 22:20:48.410275575 +0000 UTC m=+0.201394206 container died c06c2efb9f9ed9e5fb486e2807da25167dcc47ccf79169db1fb109d1b8a1a333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:20:49.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:48 smithi174 podman[42498]: 2024-03-11 22:20:48.798344471 +0000 UTC m=+0.589463103 container cleanup c06c2efb9f9ed9e5fb486e2807da25167dcc47ccf79169db1fb109d1b8a1a333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, ceph=True) 2024-03-11T22:20:49.005 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 11 22:20:48 smithi174 bash[42498]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-b 2024-03-11T22:20:49.164 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mon.b.service' 2024-03-11T22:20:49.214 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:20:49.214 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-11T22:20:49.214 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-11T22:20:49.214 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mon.c 2024-03-11T22:20:49.597 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:49 smithi183 systemd[1]: Stopping Ceph mon.c for c71f9f14-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:20:49.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:49 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-c[30883]: 2024-03-11T22:20:49.594+0000 7f909c41d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:20:49.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:49 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-c[30883]: 2024-03-11T22:20:49.594+0000 7f909c41d700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-11T22:20:49.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:49 smithi183 podman[44524]: 2024-03-11 22:20:49.778219545 +0000 UTC m=+0.366408390 container died b32fe2121d6f65c953cfe2785f69edf1b058faca64fd7c20c50876fc74fd1ae3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-03-11T22:20:50.022 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:49 smithi027 bash[60151]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-11T22:20:50.023 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:49 smithi027 bash[60151]: Writing manifest to image destination 2024-03-11T22:20:50.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:50 smithi183 podman[44524]: 2024-03-11 22:20:50.216026519 +0000 UTC m=+0.804215360 container cleanup b32fe2121d6f65c953cfe2785f69edf1b058faca64fd7c20c50876fc74fd1ae3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-11T22:20:50.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 11 22:20:50 smithi183 bash[44524]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mon-c 2024-03-11T22:20:50.414 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 podman[60151]: 2024-03-11 22:20:50.071254666 +0000 UTC m=+3.319874577 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-11T22:20:50.414 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 podman[60151]: 2024-03-11 22:20:50.169332427 +0000 UTC m=+3.417952381 container create c56b3efa8f4744115dd3331f0662c2a8ee79168ad546571451feb23829b13090 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:20:50.585 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mon.c.service' 2024-03-11T22:20:50.631 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:20:50.631 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-11T22:20:50.631 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-11T22:20:50.631 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mgr.y 2024-03-11T22:20:50.675 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: ::ffff:172.21.15.174 - - [11/Mar/2024:22:20:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:20:50.676 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 podman[60151]: 2024-03-11 22:20:50.413307057 +0000 UTC m=+3.661926967 container init c56b3efa8f4744115dd3331f0662c2a8ee79168ad546571451feb23829b13090 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:20:50.676 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 podman[60151]: 2024-03-11 22:20:50.419885179 +0000 UTC m=+3.668505072 container start c56b3efa8f4744115dd3331f0662c2a8ee79168ad546571451feb23829b13090 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:20:50.676 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.424Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-11T22:20:50.676 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.424Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-11T22:20:50.676 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.424Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-11T22:20:50.676 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T22:20:50.676 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-11T22:20:50.676 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-11T22:20:50.676 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-11T22:20:50.677 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=arp 2024-03-11T22:20:50.677 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-11T22:20:50.677 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-11T22:20:50.677 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-11T22:20:50.677 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-11T22:20:50.677 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-11T22:20:50.677 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-11T22:20:50.678 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-11T22:20:50.678 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-11T22:20:50.678 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=edac 2024-03-11T22:20:50.678 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-11T22:20:50.678 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-11T22:20:50.678 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-11T22:20:50.678 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-11T22:20:50.678 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-11T22:20:50.678 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-11T22:20:50.678 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-11T22:20:50.678 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-11T22:20:50.678 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-11T22:20:50.679 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-11T22:20:50.679 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-11T22:20:50.679 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-11T22:20:50.679 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-11T22:20:50.679 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-11T22:20:50.679 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-11T22:20:50.679 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-11T22:20:50.679 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=os 2024-03-11T22:20:50.679 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-11T22:20:50.679 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-11T22:20:50.679 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-11T22:20:50.679 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-11T22:20:50.679 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-11T22:20:50.679 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-11T22:20:50.680 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-11T22:20:50.680 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=stat 2024-03-11T22:20:50.680 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-11T22:20:50.680 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-11T22:20:50.680 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-11T22:20:50.680 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=time 2024-03-11T22:20:50.680 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-11T22:20:50.680 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.425Z caller=node_exporter.go:117 level=info collector=uname 2024-03-11T22:20:50.680 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.426Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-11T22:20:50.680 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.426Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-11T22:20:50.680 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.426Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-11T22:20:50.680 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.426Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-11T22:20:50.680 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a[60268]: ts=2024-03-11T22:20:50.426Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T22:20:50.681 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 bash[60151]: c56b3efa8f4744115dd3331f0662c2a8ee79168ad546571451feb23829b13090 2024-03-11T22:20:50.681 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:20:50 smithi027 systemd[1]: Started Ceph node-exporter.a for c71f9f14-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:20:50.995 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:50 smithi027 systemd[1]: Stopping Ceph mgr.y for c71f9f14-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:20:51.273 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:50 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y[26054]: 2024-03-11T22:20:50.992+0000 7ffa644b3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:20:51.273 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:51 smithi027 podman[60346]: 2024-03-11 22:20:51.024629367 +0000 UTC m=+0.210948116 container died bd177f900fda5bd7c3d2e597984f2aa220de753a4880efa9d489747d986c15d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-11T22:20:51.653 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:51 smithi027 podman[60346]: 2024-03-11 22:20:51.298682027 +0000 UTC m=+0.485000772 container cleanup bd177f900fda5bd7c3d2e597984f2aa220de753a4880efa9d489747d986c15d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-11T22:20:51.653 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:51 smithi027 bash[60346]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y 2024-03-11T22:20:51.653 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Mar 11 22:20:51 smithi027 podman[60366]: 2024-03-11 22:20:51.541964944 +0000 UTC m=+0.517734882 container remove bd177f900fda5bd7c3d2e597984f2aa220de753a4880efa9d489747d986c15d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-y, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-94a5397) 2024-03-11T22:20:51.668 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-11T22:20:51.709 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:20:51.709 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-11T22:20:51.709 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-11T22:20:51.709 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mgr.x 2024-03-11T22:20:52.005 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:51 smithi174 systemd[1]: Stopping Ceph mgr.x for c71f9f14-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:20:52.277 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:52 smithi174 podman[42627]: 2024-03-11 22:20:52.009146473 +0000 UTC m=+0.113573892 container died aea4c8612d5dd3874a9bb1d14f5d2d6bcc37346dd8e3af3b7bc669c5e517a979 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:20:52.537 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:52 smithi174 podman[42627]: 2024-03-11 22:20:52.276398944 +0000 UTC m=+0.380826373 container cleanup aea4c8612d5dd3874a9bb1d14f5d2d6bcc37346dd8e3af3b7bc669c5e517a979 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-11T22:20:52.537 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 11 22:20:52 smithi174 bash[42627]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-x 2024-03-11T22:20:52.686 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-11T22:20:52.722 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:20:52.723 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-11T22:20:52.723 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-11T22:20:52.723 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mgr.z 2024-03-11T22:20:53.104 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:52 smithi183 systemd[1]: Stopping Ceph mgr.z for c71f9f14-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:20:53.604 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:53 smithi183 podman[44655]: 2024-03-11 22:20:53.327753282 +0000 UTC m=+0.410655524 container died 6342b9a9365073a1b55bfc74b1185398f7b6b8b1dbc3b9c9c27d55782b274f93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-11T22:20:53.971 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:53 smithi183 podman[44655]: 2024-03-11 22:20:53.677781015 +0000 UTC m=+0.760683254 container cleanup 6342b9a9365073a1b55bfc74b1185398f7b6b8b1dbc3b9c9c27d55782b274f93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:20:53.972 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 11 22:20:53 smithi183 bash[44655]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-mgr-z 2024-03-11T22:20:54.140 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@mgr.z.service' 2024-03-11T22:20:54.174 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:20:54.175 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-11T22:20:54.175 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-11T22:20:54.175 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@osd.0 2024-03-11T22:20:54.482 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:20:54 smithi027 systemd[1]: Stopping Ceph osd.0 for c71f9f14-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:20:54.772 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:20:54 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-0[44600]: 2024-03-11T22:20:54.479+0000 7f970b81b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:20:54.773 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:20:54 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-0[44600]: 2024-03-11T22:20:54.479+0000 7f970b81b700 -1 osd.0 32 *** Got signal Terminated *** 2024-03-11T22:20:54.773 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:20:54 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-0[44600]: 2024-03-11T22:20:54.479+0000 7f970b81b700 -1 osd.0 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:20:56.505 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:20:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:20:56.164Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.27:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.27:8765: connect: connection refused" 2024-03-11T22:20:56.505 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:20:56 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:20:56.164Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.27:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.27:8765: connect: connection refused" 2024-03-11T22:21:00.174 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:20:59 smithi027 podman[60480]: 2024-03-11 22:20:59.91939558 +0000 UTC m=+5.547824574 container died 7536fc2af891534be5ad716a6c721a0d5184d957bd443a62695e5ea1c0535c3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:21:00.523 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:21:00 smithi027 podman[60480]: 2024-03-11 22:21:00.337580382 +0000 UTC m=+5.966009416 container cleanup 7536fc2af891534be5ad716a6c721a0d5184d957bd443a62695e5ea1c0535c3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:21:00.523 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:21:00 smithi027 bash[60480]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-0 2024-03-11T22:21:00.962 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:21:00 smithi027 podman[60502]: 2024-03-11 22:21:00.538630334 +0000 UTC m=+0.619402863 container remove 7536fc2af891534be5ad716a6c721a0d5184d957bd443a62695e5ea1c0535c3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-11T22:21:01.220 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:21:00 smithi027 podman[60600]: 2024-03-11 22:21:00.863729265 +0000 UTC m=+0.018154152 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:21:01.220 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:21:00 smithi027 podman[60600]: 2024-03-11 22:21:00.964324023 +0000 UTC m=+0.118748890 container create f45237de1d1eaf82a32e8cf343966fbe189e985eb04fde58849e4b59b48396c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-11T22:21:01.220 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:21:01 smithi027 podman[60600]: 2024-03-11 22:21:01.218406807 +0000 UTC m=+0.372831688 container init f45237de1d1eaf82a32e8cf343966fbe189e985eb04fde58849e4b59b48396c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-11T22:21:01.523 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:21:01 smithi027 podman[60600]: 2024-03-11 22:21:01.225349501 +0000 UTC m=+0.379774369 container start f45237de1d1eaf82a32e8cf343966fbe189e985eb04fde58849e4b59b48396c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-94a5397) 2024-03-11T22:21:01.523 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:21:01 smithi027 podman[60600]: 2024-03-11 22:21:01.300452167 +0000 UTC m=+0.454877047 container attach f45237de1d1eaf82a32e8cf343966fbe189e985eb04fde58849e4b59b48396c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-11T22:21:01.830 INFO:journalctl@ceph.osd.0.smithi027.stdout:Mar 11 22:21:01 smithi027 podman[60600]: 2024-03-11 22:21:01.532887233 +0000 UTC m=+0.687312102 container died f45237de1d1eaf82a32e8cf343966fbe189e985eb04fde58849e4b59b48396c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:21:02.188 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@osd.0.service' 2024-03-11T22:21:02.236 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:21:02.237 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-11T22:21:02.237 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-11T22:21:02.237 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@osd.1 2024-03-11T22:21:02.542 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:21:02 smithi174 systemd[1]: Stopping Ceph osd.1 for c71f9f14-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:21:03.005 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:21:02 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-1[34647]: 2024-03-11T22:21:02.540+0000 7fce80eeb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:21:03.005 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:21:02 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-1[34647]: 2024-03-11T22:21:02.540+0000 7fce80eeb700 -1 osd.1 32 *** Got signal Terminated *** 2024-03-11T22:21:03.005 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:21:02 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-1[34647]: 2024-03-11T22:21:02.540+0000 7fce80eeb700 -1 osd.1 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:21:08.215 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:21:07 smithi174 podman[42759]: 2024-03-11 22:21:07.937273782 +0000 UTC m=+5.508968889 container died d02318eccc707c76152ab4875061ca17e66c94aa72cfce477ca2175bf48aeb48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True) 2024-03-11T22:21:08.505 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:21:08 smithi174 podman[42759]: 2024-03-11 22:21:08.303586246 +0000 UTC m=+5.875281353 container cleanup d02318eccc707c76152ab4875061ca17e66c94aa72cfce477ca2175bf48aeb48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-1, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-11T22:21:08.505 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:21:08 smithi174 bash[42759]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-1 2024-03-11T22:21:08.918 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:21:08 smithi174 podman[42782]: 2024-03-11 22:21:08.571359202 +0000 UTC m=+0.632788761 container remove d02318eccc707c76152ab4875061ca17e66c94aa72cfce477ca2175bf48aeb48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-03-11T22:21:09.201 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:21:08 smithi174 podman[42875]: 2024-03-11 22:21:08.820530531 +0000 UTC m=+0.018111531 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:21:09.201 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:21:08 smithi174 podman[42875]: 2024-03-11 22:21:08.921995019 +0000 UTC m=+0.119576016 container create 38c6bfe2504e751f3fc4fea4bffa86b6b132ded1e772b28ca02dde4dba4fd27e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:21:09.201 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:21:09 smithi174 podman[42875]: 2024-03-11 22:21:09.124490502 +0000 UTC m=+0.322071503 container init 38c6bfe2504e751f3fc4fea4bffa86b6b132ded1e772b28ca02dde4dba4fd27e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:21:09.201 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:21:09 smithi174 podman[42875]: 2024-03-11 22:21:09.129645831 +0000 UTC m=+0.327226828 container start 38c6bfe2504e751f3fc4fea4bffa86b6b132ded1e772b28ca02dde4dba4fd27e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-1-deactivate, ceph=True, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-11T22:21:09.201 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:21:09 smithi174 podman[42875]: 2024-03-11 22:21:09.199178306 +0000 UTC m=+0.396759317 container attach 38c6bfe2504e751f3fc4fea4bffa86b6b132ded1e772b28ca02dde4dba4fd27e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-11T22:21:09.505 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 11 22:21:09 smithi174 podman[42875]: 2024-03-11 22:21:09.429239182 +0000 UTC m=+0.626820191 container died 38c6bfe2504e751f3fc4fea4bffa86b6b132ded1e772b28ca02dde4dba4fd27e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-03-11T22:21:10.009 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@osd.1.service' 2024-03-11T22:21:10.050 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:21:10.050 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-11T22:21:10.050 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-11T22:21:10.050 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@osd.2 2024-03-11T22:21:10.354 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:21:10 smithi183 systemd[1]: Stopping Ceph osd.2 for c71f9f14-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:21:10.756 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:21:10 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-2[36147]: 2024-03-11T22:21:10.448+0000 7fa1b9732700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:21:10.756 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:21:10 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-2[36147]: 2024-03-11T22:21:10.448+0000 7fa1b9732700 -1 osd.2 32 *** Got signal Terminated *** 2024-03-11T22:21:10.756 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:21:10 smithi183 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-2[36147]: 2024-03-11T22:21:10.448+0000 7fa1b9732700 -1 osd.2 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:21:16.213 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:21:15 smithi183 podman[44788]: 2024-03-11 22:21:15.922671559 +0000 UTC m=+5.689346299 container died 047f9f73e76fa3f10e755391abf241d5426f08e00c2640102c87f9e1ef9396f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-94a5397) 2024-03-11T22:21:16.214 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:21:16 smithi183 podman[44788]: 2024-03-11 22:21:16.211643492 +0000 UTC m=+5.978318233 container cleanup 047f9f73e76fa3f10e755391abf241d5426f08e00c2640102c87f9e1ef9396f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-2, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:21:16.604 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:21:16 smithi183 bash[44788]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-2 2024-03-11T22:21:16.604 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:21:16 smithi183 podman[44822]: 2024-03-11 22:21:16.420854549 +0000 UTC m=+0.490594992 container remove 047f9f73e76fa3f10e755391abf241d5426f08e00c2640102c87f9e1ef9396f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=squid-94a5397) 2024-03-11T22:21:17.042 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:21:16 smithi183 podman[44919]: 2024-03-11 22:21:16.649656402 +0000 UTC m=+0.018363277 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:21:17.042 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:21:16 smithi183 podman[44919]: 2024-03-11 22:21:16.7465723 +0000 UTC m=+0.115279169 container create a328aa1a826b96b11186e0a1a84f86e1e2b310670efb2ffbd8ef8dfdf7327fc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-11T22:21:17.043 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:21:16 smithi183 podman[44919]: 2024-03-11 22:21:16.92421983 +0000 UTC m=+0.292926699 container init a328aa1a826b96b11186e0a1a84f86e1e2b310670efb2ffbd8ef8dfdf7327fc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-11T22:21:17.043 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:21:16 smithi183 podman[44919]: 2024-03-11 22:21:16.929680123 +0000 UTC m=+0.298386994 container start a328aa1a826b96b11186e0a1a84f86e1e2b310670efb2ffbd8ef8dfdf7327fc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-11T22:21:17.043 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:21:17 smithi183 podman[44919]: 2024-03-11 22:21:17.040710508 +0000 UTC m=+0.409417404 container attach a328aa1a826b96b11186e0a1a84f86e1e2b310670efb2ffbd8ef8dfdf7327fc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-94a5397, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-11T22:21:17.355 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 11 22:21:17 smithi183 podman[44919]: 2024-03-11 22:21:17.22056637 +0000 UTC m=+0.589273229 container died a328aa1a826b96b11186e0a1a84f86e1e2b310670efb2ffbd8ef8dfdf7327fc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-94a5397) 2024-03-11T22:21:17.816 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@osd.2.service' 2024-03-11T22:21:17.857 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:21:17.857 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-11T22:21:17.857 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-11T22:21:17.857 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@rgw.foo.a 2024-03-11T22:21:18.177 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Mar 11 22:21:17 smithi027 systemd[1]: Stopping Ceph rgw.foo.a for c71f9f14-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:21:18.522 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Mar 11 22:21:18 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-rgw-foo-a[50680]: 2024-03-11T22:21:18.174+0000 7fbbea2ae700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:21:18.523 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Mar 11 22:21:18 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-rgw-foo-a[50680]: 2024-03-11T22:21:18.174+0000 7fbbf8b1eac0 -1 shutting down 2024-03-11T22:21:28.458 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Mar 11 22:21:28 smithi027 bash[60743]: time="2024-03-11T22:21:28Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-11T22:21:28.458 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Mar 11 22:21:28 smithi027 podman[60743]: 2024-03-11 22:21:28.205021468 +0000 UTC m=+10.136495681 container died 9cb6f13adf41e941e47b97cde65484137c7a1f4777a10501fff285f0ece0734d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-rgw-foo-a, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-11T22:21:28.773 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Mar 11 22:21:28 smithi027 podman[60743]: 2024-03-11 22:21:28.649133392 +0000 UTC m=+10.580607604 container cleanup 9cb6f13adf41e941e47b97cde65484137c7a1f4777a10501fff285f0ece0734d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-11T22:21:28.773 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Mar 11 22:21:28 smithi027 bash[60743]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-rgw-foo-a 2024-03-11T22:21:29.026 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@rgw.foo.a.service' 2024-03-11T22:21:29.066 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:21:29.066 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-11T22:21:29.067 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-11T22:21:29.067 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@prometheus.a 2024-03-11T22:21:29.364 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:21:29 smithi174 systemd[1]: Stopping Ceph prometheus.a for c71f9f14-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:21:29.637 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:21:29 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:21:29.362Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-11T22:21:29.637 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:21:29 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:21:29.362Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-11T22:21:29.637 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:21:29 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:21:29.362Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-11T22:21:29.637 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:21:29 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:21:29.362Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-11T22:21:29.637 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:21:29 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:21:29.362Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-11T22:21:29.637 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:21:29 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:21:29.362Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-11T22:21:29.637 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:21:29 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:21:29.362Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-11T22:21:29.637 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:21:29 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:21:29.362Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-11T22:21:29.637 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:21:29 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:21:29.362Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-11T22:21:29.637 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:21:29 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:21:29.362Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-11T22:21:29.638 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:21:29 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:21:29.362Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-11T22:21:29.638 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:21:29 smithi174 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a[40487]: ts=2024-03-11T22:21:29.363Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-11T22:21:29.638 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:21:29 smithi174 podman[43017]: 2024-03-11 22:21:29.372879847 +0000 UTC m=+0.114667057 container died 6253f8395a1769acc6e588f89d8c7e1e68bb7dc449a967dbae0be1d914e15c85 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:21:29.978 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:21:29 smithi174 podman[43017]: 2024-03-11 22:21:29.725599902 +0000 UTC m=+0.467387120 container cleanup 6253f8395a1769acc6e588f89d8c7e1e68bb7dc449a967dbae0be1d914e15c85 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:21:29.978 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 11 22:21:29 smithi174 bash[43017]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T22:21:30.142 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@prometheus.a.service' 2024-03-11T22:21:30.178 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:21:30.178 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-11T22:21:30.178 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T22:21:30.379 INFO:teuthology.orchestra.run.smithi027.stdout:Deleting cluster with fsid: c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:21:31.505 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:21:31 smithi027 systemd[1]: Stopping Ceph alertmanager.a for c71f9f14-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:21:31.769 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:21:31 smithi027 ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-alertmanager-a[58254]: ts=2024-03-11T22:21:31.503Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-11T22:21:31.770 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:21:31 smithi027 podman[61086]: 2024-03-11 22:21:31.524245133 +0000 UTC m=+0.134320482 container died 6e10427aa23c71a8769ef30dfd4d6afe821c5653d7773851414cab1412301267 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:21:32.023 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:21:31 smithi027 podman[61086]: 2024-03-11 22:21:31.934009112 +0000 UTC m=+0.544084503 container cleanup 6e10427aa23c71a8769ef30dfd4d6afe821c5653d7773851414cab1412301267 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:21:32.023 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:21:31 smithi027 bash[61086]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T22:21:32.402 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:21:32 smithi027 podman[61105]: 2024-03-11 22:21:32.102036844 +0000 UTC m=+0.576991655 container remove 6e10427aa23c71a8769ef30dfd4d6afe821c5653d7773851414cab1412301267 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:21:32.402 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:21:32 smithi027 podman[61105]: 2024-03-11 22:21:32.177674517 +0000 UTC m=+0.652629284 volume remove cced4a3ada25c31847832d8ce49b5d444a95672bd3b78908c6840dc773c4dc78 2024-03-11T22:21:32.773 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:21:32 smithi027 systemd[1]: Stopping Ceph node-exporter.a for c71f9f14-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:21:32.773 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:21:32 smithi027 systemd[1]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-11T22:21:32.773 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Mar 11 22:21:32 smithi027 systemd[1]: Stopped Ceph alertmanager.a for c71f9f14-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:21:33.232 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:21:32 smithi027 podman[61222]: 2024-03-11 22:21:32.877026617 +0000 UTC m=+0.108111920 container died c56b3efa8f4744115dd3331f0662c2a8ee79168ad546571451feb23829b13090 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:21:33.232 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:21:33 smithi027 podman[61222]: 2024-03-11 22:21:33.230009856 +0000 UTC m=+0.461095160 container cleanup c56b3efa8f4744115dd3331f0662c2a8ee79168ad546571451feb23829b13090 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:21:33.523 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:21:33 smithi027 bash[61222]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a 2024-03-11T22:21:33.523 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:21:33 smithi027 podman[61239]: 2024-03-11 22:21:33.456318383 +0000 UTC m=+0.583763363 container remove c56b3efa8f4744115dd3331f0662c2a8ee79168ad546571451feb23829b13090 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:21:33.523 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:21:33 smithi027 systemd[1]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T22:21:34.023 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:21:33 smithi027 systemd[1]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-11T22:21:34.023 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:21:33 smithi027 systemd[1]: Stopped Ceph node-exporter.a for c71f9f14-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:21:34.023 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Mar 11 22:21:33 smithi027 systemd[1]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@node-exporter.a.service: Consumed 1.830s CPU time. 2024-03-11T22:21:34.425 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T22:21:34.613 INFO:teuthology.orchestra.run.smithi174.stdout:Deleting cluster with fsid: c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:21:35.720 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:21:35 smithi174 systemd[1]: Stopping Ceph grafana.a for c71f9f14-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:21:35.973 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:21:35 smithi174 podman[43355]: 2024-03-11 22:21:35.718242477 +0000 UTC m=+0.118890669 container died 1e6f31f66d90bc6ae103597f0368dca3597b2e34655b8423680a34f918b0bfc2 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a, description=Ceph Grafana Container, io.buildah.version=1.32.0, vendor=Red Hat, Inc., maintainer=Paul Cuzner , com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, distribution-scope=public, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897, io.openshift.tags=base rhel8, release=239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, architecture=x86_64, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git) 2024-03-11T22:21:36.239 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:21:36 smithi174 podman[43355]: 2024-03-11 22:21:36.037026389 +0000 UTC m=+0.437674568 container cleanup 1e6f31f66d90bc6ae103597f0368dca3597b2e34655b8423680a34f918b0bfc2 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a, maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, distribution-scope=public, com.redhat.component=ubi8-container, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, name=ubi8, vendor=Red Hat, Inc., vcs-type=git, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5) 2024-03-11T22:21:36.240 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:21:36 smithi174 bash[43355]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a 2024-03-11T22:21:36.506 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:21:36 smithi174 podman[43372]: 2024-03-11 22:21:36.23788256 +0000 UTC m=+0.519149789 container remove 1e6f31f66d90bc6ae103597f0368dca3597b2e34655b8423680a34f918b0bfc2 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c71f9f14-dff4-11ee-95c8-87774f69a715-grafana-a, distribution-scope=public, maintainer=Paul Cuzner , version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, build-date=2022-04-29T11:48:11.055897, io.openshift.expose-services=, vcs-type=git, architecture=x86_64) 2024-03-11T22:21:36.506 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:21:36 smithi174 systemd[1]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T22:21:36.506 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:21:36 smithi174 systemd[1]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-11T22:21:36.506 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:21:36 smithi174 systemd[1]: Stopped Ceph grafana.a for c71f9f14-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:21:36.506 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 11 22:21:36 smithi174 systemd[1]: ceph-c71f9f14-dff4-11ee-95c8-87774f69a715@grafana.a.service: Consumed 2.660s CPU time. 2024-03-11T22:21:37.118 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T22:21:37.312 INFO:teuthology.orchestra.run.smithi183.stdout:Deleting cluster with fsid: c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:21:38.377 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:21:38.416 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:21:38.451 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:21:38.479 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-11T22:21:38.480 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591967/remote/smithi027/crash 2024-03-11T22:21:38.482 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/crash -- . 2024-03-11T22:21:38.571 INFO:teuthology.orchestra.run.smithi027.stderr:tar: /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T22:21:38.571 INFO:teuthology.orchestra.run.smithi027.stderr:tar: Error is not recoverable: exiting now 2024-03-11T22:21:38.573 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591967/remote/smithi174/crash 2024-03-11T22:21:38.574 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/crash -- . 2024-03-11T22:21:38.613 INFO:teuthology.orchestra.run.smithi174.stderr:tar: /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T22:21:38.613 INFO:teuthology.orchestra.run.smithi174.stderr:tar: Error is not recoverable: exiting now 2024-03-11T22:21:38.615 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591967/remote/smithi183/crash 2024-03-11T22:21:38.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/crash -- . 2024-03-11T22:21:38.653 INFO:teuthology.orchestra.run.smithi183.stderr:tar: /var/lib/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T22:21:38.653 INFO:teuthology.orchestra.run.smithi183.stderr:tar: Error is not recoverable: exiting now 2024-03-11T22:21:38.654 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-11T22:21:38.654 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T22:21:38.693 INFO:teuthology.orchestra.run.smithi027.stdout:2024-03-11T22:17:55.348479+0000 mon.a (mon.0) 193 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T22:21:38.694 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-11T22:21:38.694 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[SEC\]' /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T22:21:38.769 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]' /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T22:21:38.843 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[WRN\]' /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T22:21:38.917 INFO:teuthology.orchestra.run.smithi027.stdout:2024-03-11T22:17:55.348479+0000 mon.a (mon.0) 193 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T22:21:38.918 INFO:tasks.cephadm:Compressing logs... 2024-03-11T22:21:38.918 DEBUG:teuthology.orchestra.run.smithi027:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T22:21:38.960 DEBUG:teuthology.orchestra.run.smithi174:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T22:21:38.963 DEBUG:teuthology.orchestra.run.smithi183:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T22:21:38.991 INFO:teuthology.orchestra.run.smithi183.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T22:21:38.991 INFO:teuthology.orchestra.run.smithi183.stderr:: No such file or directory 2024-03-11T22:21:38.991 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T22:21:38.991 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-mgr.z.log 2024-03-11T22:21:38.991 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T22:21:38.992 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.log 2024-03-11T22:21:38.992 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-11T22:21:38.992 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.cephadm.log: 90.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T22:21:38.992 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.log: 80.4% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T22:21:38.992 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T22:21:38.992 INFO:teuthology.orchestra.run.smithi183.stderr: 89.5% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T22:21:38.992 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-11T22:21:38.992 INFO:teuthology.orchestra.run.smithi183.stderr: 84.3% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T22:21:38.997 INFO:teuthology.orchestra.run.smithi174.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T22:21:38.997 INFO:teuthology.orchestra.run.smithi174.stderr:: No such file or directory 2024-03-11T22:21:38.997 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T22:21:38.997 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-11T22:21:38.997 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-11T22:21:38.997 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T22:21:38.997 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.log 2024-03-11T22:21:38.997 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-mgr.x.log: 89.4% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T22:21:38.997 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T22:21:38.997 INFO:teuthology.orchestra.run.smithi174.stderr: 80.4% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T22:21:38.997 INFO:teuthology.orchestra.run.smithi174.stderr:gzip/var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.log: -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-11T22:21:38.997 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-volume.log: 80.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T22:21:38.997 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-mon.b.log: 84.1% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T22:21:38.999 INFO:teuthology.orchestra.run.smithi174.stderr: 90.5% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T22:21:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-11T22:21:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T22:21:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-11T22:21:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T22:21:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-11T22:21:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-11T22:21:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-11T22:21:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log: /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-mgr.y.log: 57.6% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-11T22:21:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T22:21:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.log 2024-03-11T22:21:39.000 INFO:teuthology.orchestra.run.smithi027.stderr: 89.4% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T22:21:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T22:21:39.000 INFO:teuthology.orchestra.run.smithi027.stderr: 90.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T22:21:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.log: 82.3% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T22:21:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-volume.log: 84.8% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T22:21:39.000 INFO:teuthology.orchestra.run.smithi027.stderr: 90.4% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T22:21:39.000 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-mon.c.log: 90.1% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-mgr.z.log.gz 2024-03-11T22:21:39.000 INFO:teuthology.orchestra.run.smithi183.stderr: 90.5% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T22:21:39.001 INFO:teuthology.orchestra.run.smithi174.stderr: 90.2% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-11T22:21:39.044 INFO:teuthology.orchestra.run.smithi183.stderr: 92.1% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-11T22:21:39.044 INFO:teuthology.orchestra.run.smithi174.stderr: 92.2% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-11T22:21:39.057 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-11T22:21:39.164 INFO:teuthology.orchestra.run.smithi027.stderr: 91.4% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-11T22:21:39.239 INFO:teuthology.orchestra.run.smithi183.stderr: 94.3% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-11T22:21:39.241 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-11T22:21:39.241 INFO:teuthology.orchestra.run.smithi183.stderr:real 0m0.264s 2024-03-11T22:21:39.241 INFO:teuthology.orchestra.run.smithi183.stderr:user 0m0.313s 2024-03-11T22:21:39.241 INFO:teuthology.orchestra.run.smithi183.stderr:sys 0m0.029s 2024-03-11T22:21:39.249 INFO:teuthology.orchestra.run.smithi027.stderr: 94.3% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-11T22:21:39.251 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-03-11T22:21:39.251 INFO:teuthology.orchestra.run.smithi027.stderr:real 0m0.278s 2024-03-11T22:21:39.252 INFO:teuthology.orchestra.run.smithi027.stderr:user 0m0.515s 2024-03-11T22:21:39.252 INFO:teuthology.orchestra.run.smithi027.stderr:sys 0m0.035s 2024-03-11T22:21:39.286 INFO:teuthology.orchestra.run.smithi174.stderr: 94.3% -- replaced with /var/log/ceph/c71f9f14-dff4-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-11T22:21:39.288 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-11T22:21:39.288 INFO:teuthology.orchestra.run.smithi174.stderr:real 0m0.310s 2024-03-11T22:21:39.288 INFO:teuthology.orchestra.run.smithi174.stderr:user 0m0.351s 2024-03-11T22:21:39.288 INFO:teuthology.orchestra.run.smithi174.stderr:sys 0m0.041s 2024-03-11T22:21:39.288 INFO:tasks.cephadm:Archiving logs... 2024-03-11T22:21:39.289 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591967/remote/smithi027/log 2024-03-11T22:21:39.290 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T22:21:39.416 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591967/remote/smithi174/log 2024-03-11T22:21:39.417 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T22:21:39.502 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591967/remote/smithi183/log 2024-03-11T22:21:39.503 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T22:21:39.583 INFO:tasks.cephadm:Removing cluster... 2024-03-11T22:21:39.584 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 --force 2024-03-11T22:21:39.774 INFO:teuthology.orchestra.run.smithi027.stdout:Deleting cluster with fsid: c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:21:39.895 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 --force 2024-03-11T22:21:40.082 INFO:teuthology.orchestra.run.smithi174.stdout:Deleting cluster with fsid: c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:21:40.203 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c71f9f14-dff4-11ee-95c8-87774f69a715 --force 2024-03-11T22:21:40.384 INFO:teuthology.orchestra.run.smithi183.stdout:Deleting cluster with fsid: c71f9f14-dff4-11ee-95c8-87774f69a715 2024-03-11T22:21:40.498 INFO:tasks.cephadm:Removing cephadm ... 2024-03-11T22:21:40.498 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T22:21:40.518 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T22:21:40.532 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T22:21:40.547 INFO:tasks.cephadm:Teardown complete 2024-03-11T22:21:40.548 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-11T22:21:40.559 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi027:/dev/vg_nvme/lv_1... 2024-03-11T22:21:40.559 DEBUG:teuthology.orchestra.run.smithi027:> sudo nvme disconnect -n lv_1 2024-03-11T22:21:40.705 INFO:teuthology.orchestra.run.smithi027.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-11T22:21:40.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:40.710 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi027:/dev/vg_nvme/lv_2... 2024-03-11T22:21:40.711 DEBUG:teuthology.orchestra.run.smithi027:> sudo nvme disconnect -n lv_2 2024-03-11T22:21:40.870 INFO:teuthology.orchestra.run.smithi027.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-11T22:21:40.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:40.872 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi027:/dev/vg_nvme/lv_3... 2024-03-11T22:21:40.872 DEBUG:teuthology.orchestra.run.smithi027:> sudo nvme disconnect -n lv_3 2024-03-11T22:21:41.000 INFO:teuthology.orchestra.run.smithi027.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-11T22:21:41.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:41.002 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi027:/dev/vg_nvme/lv_4... 2024-03-11T22:21:41.002 DEBUG:teuthology.orchestra.run.smithi027:> sudo nvme disconnect -n lv_4 2024-03-11T22:21:41.156 INFO:teuthology.orchestra.run.smithi027.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-11T22:21:41.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:41.158 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-11T22:21:41.158 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/scratch_devs 2024-03-11T22:21:41.192 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi174:/dev/vg_nvme/lv_1... 2024-03-11T22:21:41.192 DEBUG:teuthology.orchestra.run.smithi174:> sudo nvme disconnect -n lv_1 2024-03-11T22:21:41.346 INFO:teuthology.orchestra.run.smithi174.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-11T22:21:41.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:41.348 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi174:/dev/vg_nvme/lv_2... 2024-03-11T22:21:41.348 DEBUG:teuthology.orchestra.run.smithi174:> sudo nvme disconnect -n lv_2 2024-03-11T22:21:41.484 INFO:teuthology.orchestra.run.smithi174.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-11T22:21:41.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:41.486 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi174:/dev/vg_nvme/lv_3... 2024-03-11T22:21:41.486 DEBUG:teuthology.orchestra.run.smithi174:> sudo nvme disconnect -n lv_3 2024-03-11T22:21:41.624 INFO:teuthology.orchestra.run.smithi174.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-11T22:21:41.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:41.626 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi174:/dev/vg_nvme/lv_4... 2024-03-11T22:21:41.626 DEBUG:teuthology.orchestra.run.smithi174:> sudo nvme disconnect -n lv_4 2024-03-11T22:21:41.774 INFO:teuthology.orchestra.run.smithi174.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-11T22:21:41.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:41.776 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-11T22:21:41.776 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/scratch_devs 2024-03-11T22:21:41.806 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi183:/dev/vg_nvme/lv_1... 2024-03-11T22:21:41.806 DEBUG:teuthology.orchestra.run.smithi183:> sudo nvme disconnect -n lv_1 2024-03-11T22:21:41.959 INFO:teuthology.orchestra.run.smithi183.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-11T22:21:41.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:41.960 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi183:/dev/vg_nvme/lv_2... 2024-03-11T22:21:41.960 DEBUG:teuthology.orchestra.run.smithi183:> sudo nvme disconnect -n lv_2 2024-03-11T22:21:42.100 INFO:teuthology.orchestra.run.smithi183.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-11T22:21:42.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:42.102 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi183:/dev/vg_nvme/lv_3... 2024-03-11T22:21:42.102 DEBUG:teuthology.orchestra.run.smithi183:> sudo nvme disconnect -n lv_3 2024-03-11T22:21:42.240 INFO:teuthology.orchestra.run.smithi183.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-11T22:21:42.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:42.242 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi183:/dev/vg_nvme/lv_4... 2024-03-11T22:21:42.242 DEBUG:teuthology.orchestra.run.smithi183:> sudo nvme disconnect -n lv_4 2024-03-11T22:21:42.374 INFO:teuthology.orchestra.run.smithi183.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-11T22:21:42.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:42.376 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-11T22:21:42.376 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/scratch_devs 2024-03-11T22:21:42.403 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-11T22:21:42.414 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-11T22:21:42.414 DEBUG:teuthology.orchestra.run.smithi027:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:21:42.417 DEBUG:teuthology.orchestra.run.smithi174:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:21:42.419 DEBUG:teuthology.orchestra.run.smithi183:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:21:42.431 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:21:42.433 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:21:42.437 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:21:42.437 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-03-11T22:21:42.437 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 30 +5544us[+5544us] +/- 124ms 2024-03-11T22:21:42.437 INFO:teuthology.orchestra.run.smithi027.stdout:^* hv02.front.sepia.ceph.com 3 6 377 31 -1695us[-1702us] +/- 53ms 2024-03-11T22:21:42.437 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 -673us[ -673us] +/- 69ms 2024-03-11T22:21:42.437 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:21:42.439 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:21:42.439 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-03-11T22:21:42.439 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 30 +5567us[+5567us] +/- 124ms 2024-03-11T22:21:42.439 INFO:teuthology.orchestra.run.smithi174.stdout:^* hv02.front.sepia.ceph.com 3 6 377 32 -1704us[-1708us] +/- 53ms 2024-03-11T22:21:42.439 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 -682us[ -682us] +/- 69ms 2024-03-11T22:21:42.439 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:21:42.457 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:21:42.463 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:21:42.463 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-11T22:21:42.463 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 29 +5556us[+5556us] +/- 124ms 2024-03-11T22:21:42.463 INFO:teuthology.orchestra.run.smithi183.stdout:^* hv02.front.sepia.ceph.com 3 6 377 32 -1684us[-1655us] +/- 53ms 2024-03-11T22:21:42.463 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 -678us[ -678us] +/- 69ms 2024-03-11T22:21:42.463 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:21:42.464 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-11T22:21:42.472 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-11T22:21:42.473 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-11T22:21:42.494 DEBUG:teuthology.orchestra.run.smithi027:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T22:21:42.582 DEBUG:teuthology.orchestra.run.smithi174:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T22:21:42.657 DEBUG:teuthology.orchestra.run.smithi183:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T22:21:42.725 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:21:42.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:42.762 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:21:42.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:42.795 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:21:42.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:42.825 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-11T22:21:42.834 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-11T22:21:42.869 INFO:teuthology.task.internal:Duration was 687.560036 seconds 2024-03-11T22:21:42.870 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-11T22:21:42.903 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-11T22:21:42.904 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T22:21:42.906 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T22:21:42.909 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T22:21:42.951 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:21:42.953 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:21:42.958 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:21:43.390 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-11T22:21:43.390 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi027.front.sepia.ceph.com 2024-03-11T22:21:43.390 DEBUG:teuthology.orchestra.run.smithi027:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T22:21:43.412 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi174.front.sepia.ceph.com 2024-03-11T22:21:43.413 DEBUG:teuthology.orchestra.run.smithi174:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T22:21:43.442 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi183.front.sepia.ceph.com 2024-03-11T22:21:43.442 DEBUG:teuthology.orchestra.run.smithi183:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T22:21:43.469 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-11T22:21:43.469 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T22:21:43.471 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T22:21:43.484 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T22:21:43.536 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-11T22:21:43.536 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T22:21:43.546 DEBUG:teuthology.orchestra.run.smithi174:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T22:21:43.552 DEBUG:teuthology.orchestra.run.smithi183:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T22:21:43.743 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-11T22:21:43.752 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-11T22:21:43.753 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T22:21:43.783 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T22:21:43.810 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T22:21:43.837 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-11T22:21:43.847 DEBUG:teuthology.orchestra.run.smithi027:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:21:43.849 DEBUG:teuthology.orchestra.run.smithi174:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:21:43.853 DEBUG:teuthology.orchestra.run.smithi183:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:21:43.876 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = core 2024-03-11T22:21:43.876 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = core 2024-03-11T22:21:43.903 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = core 2024-03-11T22:21:43.938 DEBUG:teuthology.orchestra.run.smithi027:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:21:43.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:43.954 DEBUG:teuthology.orchestra.run.smithi174:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:21:43.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:43.968 DEBUG:teuthology.orchestra.run.smithi183:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:21:43.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:43.992 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-11T22:21:44.002 INFO:teuthology.task.internal:Transferring archived files... 2024-03-11T22:21:44.002 DEBUG:teuthology.misc:Transferring archived files from smithi027:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591967/remote/smithi027 2024-03-11T22:21:44.003 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T22:21:44.045 DEBUG:teuthology.misc:Transferring archived files from smithi174:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591967/remote/smithi174 2024-03-11T22:21:44.045 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T22:21:44.080 DEBUG:teuthology.misc:Transferring archived files from smithi183:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591967/remote/smithi183 2024-03-11T22:21:44.080 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T22:21:44.116 INFO:teuthology.task.internal:Removing archive directory... 2024-03-11T22:21:44.116 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T22:21:44.118 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T22:21:44.120 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T22:21:44.165 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-11T22:21:44.175 INFO:teuthology.task.internal:Not uploading archives. 2024-03-11T22:21:44.175 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-11T22:21:44.211 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-11T22:21:44.211 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T22:21:44.214 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T22:21:44.216 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T22:21:44.226 INFO:teuthology.orchestra.run.smithi027.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 22:21 /home/ubuntu/cephtest 2024-03-11T22:21:44.228 INFO:teuthology.orchestra.run.smithi174.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 22:21 /home/ubuntu/cephtest 2024-03-11T22:21:44.230 INFO:teuthology.orchestra.run.smithi183.stdout: 266004 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 22:21 /home/ubuntu/cephtest 2024-03-11T22:21:44.231 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-11T22:21:44.245 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-11T22:21:44.321 INFO:teuthology.nuke:Checking targets against current locks 2024-03-11T22:21:44.347 DEBUG:teuthology.nuke:shortname: smithi027 2024-03-11T22:21:44.348 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:21:44.374 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591967', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:57:56.790260', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA4R4hiLP6CWK8D+CcttU1Rmgt5UhEor2Zo2LA3JP1z2cTomjDnn8/EjFiVnSVuVQ7PCXs7yOnMCyEfTsN/l7tc='} 2024-03-11T22:21:44.378 DEBUG:teuthology.nuke:shortname: smithi174 2024-03-11T22:21:44.378 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:21:44.398 DEBUG:teuthology.nuke:shortname: smithi183 2024-03-11T22:21:44.398 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:21:44.402 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591967', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:57:56.792695', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI3ptjM9LOuwQkz+C+zsIjrpORomAcZIktI+ec1eN/X4K3amWAUhGhL2onb3vwNvwQyVKogLHjx1bXYOCdQHEjM='} 2024-03-11T22:21:44.418 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591967', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:57:56.791882', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ9MaOw7ifKmwoCA9PXLsbUa415InpwWbpwXyvjlVtLGEJmbsS5yh5XZqxpJpCO/1HWyTdjpnD6sfTQn7yDgN6A='} 2024-03-11T22:21:44.429 INFO:teuthology.orchestra.console.smithi027:Power off 2024-03-11T22:21:44.430 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T22:21:44.454 DEBUG:teuthology.orchestra.console.smithi027:power off output: Chassis Power Control: Down/Off 2024-03-11T22:21:44.455 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:21:44.468 INFO:teuthology.orchestra.console.smithi174:Power off 2024-03-11T22:21:44.468 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T22:21:44.481 INFO:teuthology.orchestra.console.smithi183:Power off 2024-03-11T22:21:44.482 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T22:21:44.489 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-03-11T22:21:44.492 DEBUG:teuthology.orchestra.console.smithi174:power off output: Chassis Power Control: Down/Off 2024-03-11T22:21:44.493 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:21:44.504 DEBUG:teuthology.orchestra.console.smithi183:power off output: Chassis Power Control: Down/Off 2024-03-11T22:21:44.504 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:21:44.514 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2024-03-11T22:21:44.529 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-11T22:21:48.490 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:21:48.515 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:21:48.530 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:21:48.601 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-03-11T22:21:48.624 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2024-03-11T22:21:48.638 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-11T22:21:52.605 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:21:52.625 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:21:52.639 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:21:52.714 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is off 2024-03-11T22:21:52.734 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is off 2024-03-11T22:21:52.747 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-11T22:21:52.816 INFO:teuthology.orchestra.console.smithi027:Power off completed 2024-03-11T22:21:52.835 INFO:teuthology.orchestra.console.smithi174:Power off completed 2024-03-11T22:21:56.751 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:21:56.860 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is off 2024-03-11T22:21:56.961 INFO:teuthology.orchestra.console.smithi183:Power off completed 2024-03-11T22:21:57.065 INFO:teuthology.run:Summary data: description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} duration: 687.5600357055664 failure_reason: '"2024-03-11T22:17:55.348479+0000 mon.a (mon.0) 193 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-11T22:21:57.065 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T22:21:57.140 INFO:teuthology.run:FAIL