2024-03-26T01:44:29.069 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-26T01:44:29.069 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621927 --verbose --name teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi --owner scheduled_teuthology@teuthology --description orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} -- /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621927/orig.config.yaml 2024-03-26T01:44:29.098 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T01:44:29.177 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621927 branch: squid description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} email: ceph-qa@ceph.com first_in_suite: false job_id: '7621927' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.1 - client.1 - prometheus.a - grafana.a - node-exporter.b - - mon.c - mgr.z - osd.2 - client.2 - node-exporter.c seed: 1822 sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a sleep_before_teardown: 0 subset: 53/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de targets: smithi003.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHu32xGMBqAWvAJXCYWtSJyY3WoRcqOMPWX/KkOfrndVMku6Z42iwzgidh532QIh8HHdSHr1op51fO/tF68lt3M= smithi055.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPxwSXOgZgN7upLALulxHsKZY0dvBNg4rRvp3c2TPaCOIzHvdCHGkYTSgMLXT3dxjX94G/D8TiwKLT0eVjD7XLE= smithi066.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBADvA3mJIbyPUTGnJwZlNnBhoxmejaDAjEo1Kw5H5arYF5+cVTZlYPnORJi2jTlzc0yAVSkY3iiJPrDvuKTg26M= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_21:08:11 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-26T01:44:29.177 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa; will attempt to use it 2024-03-26T01:44:29.178 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa/tasks 2024-03-26T01:44:29.178 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-26T01:44:29.179 INFO:teuthology.task.internal:Checking packages... 2024-03-26T01:44:29.205 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5e70124b46c2a671efcaae4690cd9ea72670710a' 2024-03-26T01:44:29.205 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T01:44:29.205 INFO:teuthology.packaging:ref: None 2024-03-26T01:44:29.205 INFO:teuthology.packaging:tag: None 2024-03-26T01:44:29.205 INFO:teuthology.packaging:branch: squid 2024-03-26T01:44:29.205 INFO:teuthology.packaging:sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:44:29.206 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-26T01:44:29.375 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1863.g9f04c91b 2024-03-26T01:44:29.377 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-26T01:44:29.383 INFO:teuthology.task.internal:no buildpackages task found 2024-03-26T01:44:29.383 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-26T01:44:29.390 INFO:teuthology.task.internal:Saving configuration 2024-03-26T01:44:29.402 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-26T01:44:29.408 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T01:44:29.431 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621927', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 01:37:45.880175', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHu32xGMBqAWvAJXCYWtSJyY3WoRcqOMPWX/KkOfrndVMku6Z42iwzgidh532QIh8HHdSHr1op51fO/tF68lt3M='} 2024-03-26T01:44:29.451 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621927', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 01:37:45.881190', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPxwSXOgZgN7upLALulxHsKZY0dvBNg4rRvp3c2TPaCOIzHvdCHGkYTSgMLXT3dxjX94G/D8TiwKLT0eVjD7XLE='} 2024-03-26T01:44:29.471 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi066.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621927', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 01:37:45.878916', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBADvA3mJIbyPUTGnJwZlNnBhoxmejaDAjEo1Kw5H5arYF5+cVTZlYPnORJi2jTlzc0yAVSkY3iiJPrDvuKTg26M='} 2024-03-26T01:44:29.471 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-26T01:44:29.477 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-26T01:44:29.477 INFO:teuthology.task.internal:roles: ubuntu@smithi055.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.1', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-26T01:44:29.477 INFO:teuthology.task.internal:roles: ubuntu@smithi066.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'osd.2', 'client.2', 'node-exporter.c'] 2024-03-26T01:44:29.477 INFO:teuthology.run_tasks:Running task console_log... 2024-03-26T01:44:29.575 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd8f5b7cf70>, signals=[15]) 2024-03-26T01:44:29.576 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-26T01:44:29.584 INFO:teuthology.task.internal:Opening connections... 2024-03-26T01:44:29.584 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi003.front.sepia.ceph.com 2024-03-26T01:44:29.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:44:29.699 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi055.front.sepia.ceph.com 2024-03-26T01:44:29.700 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:44:29.808 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi066.front.sepia.ceph.com 2024-03-26T01:44:29.809 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:44:29.910 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-26T01:44:29.919 DEBUG:teuthology.orchestra.run.smithi003:> uname -m 2024-03-26T01:44:29.938 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2024-03-26T01:44:29.938 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release 2024-03-26T01:44:29.996 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="CentOS Stream" 2024-03-26T01:44:29.996 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="9" 2024-03-26T01:44:29.996 INFO:teuthology.orchestra.run.smithi003.stdout:ID="centos" 2024-03-26T01:44:29.996 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="rhel fedora" 2024-03-26T01:44:29.996 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="9" 2024-03-26T01:44:29.996 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el9" 2024-03-26T01:44:29.996 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T01:44:29.996 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31" 2024-03-26T01:44:29.997 INFO:teuthology.orchestra.run.smithi003.stdout:LOGO="fedora-logo-icon" 2024-03-26T01:44:29.997 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T01:44:29.997 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://centos.org/" 2024-03-26T01:44:29.997 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T01:44:29.997 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T01:44:29.997 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T01:44:29.997 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2024-03-26T01:44:30.023 DEBUG:teuthology.orchestra.run.smithi055:> uname -m 2024-03-26T01:44:30.042 INFO:teuthology.orchestra.run.smithi055.stdout:x86_64 2024-03-26T01:44:30.042 DEBUG:teuthology.orchestra.run.smithi055:> cat /etc/os-release 2024-03-26T01:44:30.100 INFO:teuthology.orchestra.run.smithi055.stdout:NAME="CentOS Stream" 2024-03-26T01:44:30.100 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION="9" 2024-03-26T01:44:30.100 INFO:teuthology.orchestra.run.smithi055.stdout:ID="centos" 2024-03-26T01:44:30.100 INFO:teuthology.orchestra.run.smithi055.stdout:ID_LIKE="rhel fedora" 2024-03-26T01:44:30.100 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION_ID="9" 2024-03-26T01:44:30.100 INFO:teuthology.orchestra.run.smithi055.stdout:PLATFORM_ID="platform:el9" 2024-03-26T01:44:30.100 INFO:teuthology.orchestra.run.smithi055.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T01:44:30.100 INFO:teuthology.orchestra.run.smithi055.stdout:ANSI_COLOR="0;31" 2024-03-26T01:44:30.100 INFO:teuthology.orchestra.run.smithi055.stdout:LOGO="fedora-logo-icon" 2024-03-26T01:44:30.101 INFO:teuthology.orchestra.run.smithi055.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T01:44:30.101 INFO:teuthology.orchestra.run.smithi055.stdout:HOME_URL="https://centos.org/" 2024-03-26T01:44:30.101 INFO:teuthology.orchestra.run.smithi055.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T01:44:30.101 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T01:44:30.101 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T01:44:30.101 INFO:teuthology.lock.ops:Updating smithi055.front.sepia.ceph.com on lock server 2024-03-26T01:44:30.123 DEBUG:teuthology.orchestra.run.smithi066:> uname -m 2024-03-26T01:44:30.141 INFO:teuthology.orchestra.run.smithi066.stdout:x86_64 2024-03-26T01:44:30.141 DEBUG:teuthology.orchestra.run.smithi066:> cat /etc/os-release 2024-03-26T01:44:30.202 INFO:teuthology.orchestra.run.smithi066.stdout:NAME="CentOS Stream" 2024-03-26T01:44:30.202 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION="9" 2024-03-26T01:44:30.202 INFO:teuthology.orchestra.run.smithi066.stdout:ID="centos" 2024-03-26T01:44:30.202 INFO:teuthology.orchestra.run.smithi066.stdout:ID_LIKE="rhel fedora" 2024-03-26T01:44:30.202 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION_ID="9" 2024-03-26T01:44:30.202 INFO:teuthology.orchestra.run.smithi066.stdout:PLATFORM_ID="platform:el9" 2024-03-26T01:44:30.202 INFO:teuthology.orchestra.run.smithi066.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T01:44:30.202 INFO:teuthology.orchestra.run.smithi066.stdout:ANSI_COLOR="0;31" 2024-03-26T01:44:30.202 INFO:teuthology.orchestra.run.smithi066.stdout:LOGO="fedora-logo-icon" 2024-03-26T01:44:30.202 INFO:teuthology.orchestra.run.smithi066.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T01:44:30.202 INFO:teuthology.orchestra.run.smithi066.stdout:HOME_URL="https://centos.org/" 2024-03-26T01:44:30.203 INFO:teuthology.orchestra.run.smithi066.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T01:44:30.203 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T01:44:30.203 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T01:44:30.203 INFO:teuthology.lock.ops:Updating smithi066.front.sepia.ceph.com on lock server 2024-03-26T01:44:30.223 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-26T01:44:30.233 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-26T01:44:30.238 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-26T01:44:30.238 DEBUG:teuthology.orchestra.run.smithi003:> test '!' -e /home/ubuntu/cephtest 2024-03-26T01:44:30.241 DEBUG:teuthology.orchestra.run.smithi055:> test '!' -e /home/ubuntu/cephtest 2024-03-26T01:44:30.243 DEBUG:teuthology.orchestra.run.smithi066:> test '!' -e /home/ubuntu/cephtest 2024-03-26T01:44:30.261 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-26T01:44:30.268 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-26T01:44:30.268 DEBUG:teuthology.orchestra.run.smithi003:> test -z $(ls -A /var/lib/ceph) 2024-03-26T01:44:30.299 DEBUG:teuthology.orchestra.run.smithi055:> test -z $(ls -A /var/lib/ceph) 2024-03-26T01:44:30.301 DEBUG:teuthology.orchestra.run.smithi066:> test -z $(ls -A /var/lib/ceph) 2024-03-26T01:44:30.327 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-26T01:44:30.389 INFO:teuthology.run_tasks:Running task kernel... 2024-03-26T01:44:30.403 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-26T01:44:30.403 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-26T01:44:30.403 DEBUG:teuthology.orchestra.run.smithi003:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T01:44:30.404 DEBUG:teuthology.orchestra.run.smithi055:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T01:44:30.404 DEBUG:teuthology.orchestra.run.smithi066:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T01:44:30.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T01:44:30.421 DEBUG:teuthology.orchestra.run.smithi066:> uname -r 2024-03-26T01:44:30.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T01:44:30.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T01:44:30.422 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2024-03-26T01:44:30.423 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-03-26T01:44:30.481 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-361.el9.x86_64 2024-03-26T01:44:30.481 INFO:teuthology.orchestra.run.smithi055.stdout:5.14.0-361.el9.x86_64 2024-03-26T01:44:30.481 INFO:teuthology.task.kernel:Running kernel on smithi003: 5.14.0-361.el9.x86_64 2024-03-26T01:44:30.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum install -y kernel 2024-03-26T01:44:30.481 INFO:teuthology.task.kernel:Running kernel on smithi055: 5.14.0-361.el9.x86_64 2024-03-26T01:44:30.482 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum install -y kernel 2024-03-26T01:44:30.483 INFO:teuthology.orchestra.run.smithi066.stdout:5.14.0-361.el9.x86_64 2024-03-26T01:44:30.483 INFO:teuthology.task.kernel:Running kernel on smithi066: 5.14.0-361.el9.x86_64 2024-03-26T01:44:30.483 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum install -y kernel 2024-03-26T01:44:32.616 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-26T01:44:32.700 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-26T01:44:32.805 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-26T01:44:34.083 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-26T01:44:34.175 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-26T01:44:34.230 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-26T01:44:34.941 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-26T01:44:35.462 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - BaseOS 9.1 MB/s | 8.0 MB 00:00 2024-03-26T01:44:38.549 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.0 MB 00:04 2024-03-26T01:44:40.592 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-26T01:44:44.705 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - AppStream 2.3 MB/s | 19 MB 00:08 2024-03-26T01:44:46.492 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.1 MB 00:02 2024-03-26T01:44:47.174 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - AppStream 1.7 MB/s | 19 MB 00:11 2024-03-26T01:44:48.268 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 16 kB 00:00 2024-03-26T01:44:49.561 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-26T01:44:49.599 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 6.1 MB 00:01 2024-03-26T01:44:51.098 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 16 kB 00:00 2024-03-26T01:44:53.069 INFO:teuthology.orchestra.run.smithi055.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-26T01:44:54.686 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 49 kB/s | 1.7 kB 00:00 2024-03-26T01:44:56.943 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T01:44:56.943 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T01:44:56.990 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - CRB 1.0 MB/s | 6.1 MB 00:06 2024-03-26T01:44:57.041 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-26T01:44:57.049 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-26T01:44:57.049 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2024-03-26T01:44:57.050 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-26T01:44:57.050 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-26T01:44:57.050 INFO:teuthology.orchestra.run.smithi003.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T01:44:57.050 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-26T01:44:57.050 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T01:44:57.050 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T01:44:57.050 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T01:44:57.050 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:44:57.050 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-26T01:44:57.050 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-26T01:44:57.050 INFO:teuthology.orchestra.run.smithi003.stdout:Install 4 Packages 2024-03-26T01:44:57.050 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:44:57.051 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 76 M 2024-03-26T01:44:57.051 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 123 M 2024-03-26T01:44:57.051 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-26T01:44:57.812 INFO:teuthology.orchestra.run.smithi003.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 160 kB/s | 66 kB 00:00 2024-03-26T01:44:58.142 INFO:teuthology.orchestra.run.smithi055.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-03-26T01:44:58.583 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 16 kB 00:00 2024-03-26T01:44:59.402 INFO:teuthology.orchestra.run.smithi066.stdout:Extra Packages for Enterprise Linux 29 MB/s | 21 MB 00:00 2024-03-26T01:44:59.671 INFO:teuthology.orchestra.run.smithi003.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 6.6 MB/s | 15 MB 00:02 2024-03-26T01:45:00.348 INFO:teuthology.orchestra.run.smithi055.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T01:45:00.349 INFO:teuthology.orchestra.run.smithi055.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T01:45:00.445 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T01:45:00.453 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T01:45:00.453 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2024-03-26T01:45:00.453 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T01:45:00.453 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-26T01:45:00.453 INFO:teuthology.orchestra.run.smithi055.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T01:45:00.453 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-26T01:45:00.453 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T01:45:00.454 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T01:45:00.454 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T01:45:00.454 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T01:45:00.454 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-26T01:45:00.454 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T01:45:00.454 INFO:teuthology.orchestra.run.smithi055.stdout:Install 4 Packages 2024-03-26T01:45:00.454 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T01:45:00.454 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 76 M 2024-03-26T01:45:00.455 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 123 M 2024-03-26T01:45:00.455 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-26T01:45:00.860 INFO:teuthology.orchestra.run.smithi055.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 454 kB/s | 66 kB 00:00 2024-03-26T01:45:02.901 INFO:teuthology.orchestra.run.smithi055.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 6.8 MB/s | 15 MB 00:02 2024-03-26T01:45:03.918 INFO:teuthology.orchestra.run.smithi055.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 9.0 MB/s | 27 MB 00:03 2024-03-26T01:45:04.585 INFO:teuthology.orchestra.run.smithi066.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-03-26T01:45:05.702 INFO:teuthology.orchestra.run.smithi055.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 6.7 MB/s | 33 MB 00:04 2024-03-26T01:45:05.702 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:45:05.702 INFO:teuthology.orchestra.run.smithi055.stdout:Total 14 MB/s | 76 MB 00:05 2024-03-26T01:45:06.391 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-26T01:45:06.836 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-26T01:45:06.836 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-26T01:45:06.942 INFO:teuthology.orchestra.run.smithi066.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T01:45:06.942 INFO:teuthology.orchestra.run.smithi066.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T01:45:07.036 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T01:45:07.049 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T01:45:07.049 INFO:teuthology.orchestra.run.smithi066.stdout: Package Architecture Version Repository Size 2024-03-26T01:45:07.049 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T01:45:07.049 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-26T01:45:07.049 INFO:teuthology.orchestra.run.smithi066.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T01:45:07.049 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-26T01:45:07.049 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T01:45:07.049 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T01:45:07.049 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T01:45:07.049 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T01:45:07.049 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-26T01:45:07.050 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T01:45:07.050 INFO:teuthology.orchestra.run.smithi066.stdout:Install 4 Packages 2024-03-26T01:45:07.050 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T01:45:07.050 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 76 M 2024-03-26T01:45:07.050 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 123 M 2024-03-26T01:45:07.050 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-26T01:45:07.253 INFO:teuthology.orchestra.run.smithi066.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 457 kB/s | 66 kB 00:00 2024-03-26T01:45:07.909 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-26T01:45:07.909 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-26T01:45:09.461 INFO:teuthology.orchestra.run.smithi066.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 6.3 MB/s | 15 MB 00:02 2024-03-26T01:45:10.170 INFO:teuthology.orchestra.run.smithi066.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 9.4 MB/s | 27 MB 00:02 2024-03-26T01:45:10.516 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-26T01:45:11.135 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T01:45:11.227 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:45:11.503 INFO:teuthology.orchestra.run.smithi066.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 7.6 MB/s | 33 MB 00:04 2024-03-26T01:45:11.503 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:45:11.503 INFO:teuthology.orchestra.run.smithi066.stdout:Total 17 MB/s | 76 MB 00:04 2024-03-26T01:45:12.202 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-26T01:45:12.613 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-26T01:45:12.613 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-26T01:45:13.649 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-26T01:45:13.650 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-26T01:45:13.786 INFO:teuthology.orchestra.run.smithi003.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 1.7 MB/s | 27 MB 00:15 2024-03-26T01:45:14.100 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:45:14.240 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:45:14.794 INFO:teuthology.orchestra.run.smithi003.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 1.9 MB/s | 33 MB 00:17 2024-03-26T01:45:14.795 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:45:14.795 INFO:teuthology.orchestra.run.smithi003.stdout:Total 4.3 MB/s | 76 MB 00:17 2024-03-26T01:45:15.487 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-26T01:45:15.915 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-26T01:45:15.915 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-26T01:45:16.248 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-26T01:45:16.888 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T01:45:16.940 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-26T01:45:16.940 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-26T01:45:16.995 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:45:19.308 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:45:19.340 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:45:19.544 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-26T01:45:19.890 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:45:20.021 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:45:20.196 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T01:45:20.288 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:45:23.139 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:45:23.290 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:45:24.438 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:45:25.092 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:45:25.138 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:45:28.364 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:45:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:45:30.253 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:45:33.503 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:45:55.077 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:45:55.078 INFO:teuthology.orchestra.run.smithi055.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T01:45:55.078 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T01:45:55.122 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:45:55.534 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:45:55.535 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T01:45:55.535 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:45:55.535 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:45:55.902 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:45:55.902 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T01:45:55.902 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-26T01:45:55.902 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T01:45:55.903 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T01:45:55.903 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T01:45:55.903 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T01:45:55.903 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T01:45:55.903 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T01:45:56.219 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T01:45:56.322 INFO:teuthology.orchestra.run.smithi055.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T01:45:56.322 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T01:45:56.323 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T01:45:56.323 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T01:45:56.323 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T01:45:56.323 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-26T01:45:56.323 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi055.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T01:45:56.323 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T01:45:56.406 INFO:teuthology.orchestra.run.smithi055.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T01:45:56.407 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T01:45:56.407 DEBUG:teuthology.orchestra.run.smithi055:> sudo rpm -qi grub2-tools 2024-03-26T01:45:56.451 INFO:teuthology.orchestra.run.smithi055.stdout:Name : grub2-tools 2024-03-26T01:45:56.451 INFO:teuthology.orchestra.run.smithi055.stdout:Epoch : 1 2024-03-26T01:45:56.451 INFO:teuthology.orchestra.run.smithi055.stdout:Version : 2.06 2024-03-26T01:45:56.451 INFO:teuthology.orchestra.run.smithi055.stdout:Release : 68.el9 2024-03-26T01:45:56.451 INFO:teuthology.orchestra.run.smithi055.stdout:Architecture: x86_64 2024-03-26T01:45:56.451 INFO:teuthology.orchestra.run.smithi055.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T01:45:56.451 INFO:teuthology.orchestra.run.smithi055.stdout:Group : Unspecified 2024-03-26T01:45:56.451 INFO:teuthology.orchestra.run.smithi055.stdout:Size : 8264254 2024-03-26T01:45:56.452 INFO:teuthology.orchestra.run.smithi055.stdout:License : GPLv3+ 2024-03-26T01:45:56.452 INFO:teuthology.orchestra.run.smithi055.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T01:45:56.452 INFO:teuthology.orchestra.run.smithi055.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T01:45:56.452 INFO:teuthology.orchestra.run.smithi055.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T01:45:56.452 INFO:teuthology.orchestra.run.smithi055.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T01:45:56.452 INFO:teuthology.orchestra.run.smithi055.stdout:Packager : builder@centos.org 2024-03-26T01:45:56.452 INFO:teuthology.orchestra.run.smithi055.stdout:Vendor : CentOS 2024-03-26T01:45:56.452 INFO:teuthology.orchestra.run.smithi055.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T01:45:56.452 INFO:teuthology.orchestra.run.smithi055.stdout:Summary : Support tools for GRUB. 2024-03-26T01:45:56.452 INFO:teuthology.orchestra.run.smithi055.stdout:Description : 2024-03-26T01:45:56.452 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T01:45:56.452 INFO:teuthology.orchestra.run.smithi055.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T01:45:56.453 INFO:teuthology.orchestra.run.smithi055.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T01:45:56.453 INFO:teuthology.orchestra.run.smithi055.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T01:45:56.453 INFO:teuthology.orchestra.run.smithi055.stdout:hardware devices. 2024-03-26T01:45:56.453 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T01:45:56.453 INFO:teuthology.orchestra.run.smithi055.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T01:45:56.454 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T01:45:56.454 INFO:teuthology.task.kernel:Updating grub on smithi055 to boot 5.14.0-432.el9.x86_64 2024-03-26T01:45:56.454 DEBUG:teuthology.orchestra.run.smithi055:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T01:45:57.106 INFO:teuthology.orchestra.run.smithi055.stderr:Generating grub configuration file ... 2024-03-26T01:45:58.587 INFO:teuthology.orchestra.run.smithi055.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T01:45:58.633 INFO:teuthology.orchestra.run.smithi055.stderr:done 2024-03-26T01:45:58.635 DEBUG:teuthology.orchestra.run.smithi055:> mktemp 2024-03-26T01:45:58.651 INFO:teuthology.orchestra.run.smithi055.stdout:/tmp/tmp.I7jjBtT7Db 2024-03-26T01:45:58.651 DEBUG:teuthology.orchestra.run.smithi055:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.I7jjBtT7Db 2024-03-26T01:45:58.735 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0666 /tmp/tmp.I7jjBtT7Db 2024-03-26T01:45:58.882 DEBUG:teuthology.orchestra.remote:smithi055:/tmp/tmp.I7jjBtT7Db is 6KB 2024-03-26T01:45:58.931 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.I7jjBtT7Db 2024-03-26T01:45:58.946 DEBUG:teuthology.orchestra.run.smithi055:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T01:45:59.015 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T01:45:59.015 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T01:45:59.015 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T01:45:59.015 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T01:45:59.016 DEBUG:teuthology.orchestra.run.smithi055:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T01:45:59.172 DEBUG:teuthology.orchestra.run.smithi055:> sudo shutdown -r now 2024-03-26T01:46:00.889 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:46:00.889 INFO:teuthology.orchestra.run.smithi066.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T01:46:00.889 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T01:46:00.935 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:46:01.255 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:46:01.255 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T01:46:01.255 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:46:01.255 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:46:01.676 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:46:01.676 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T01:46:01.677 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-26T01:46:01.677 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T01:46:01.677 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T01:46:01.677 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T01:46:01.677 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T01:46:01.677 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T01:46:01.677 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T01:46:01.984 DEBUG:teuthology.orchestra.run.smithi066:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T01:46:02.102 INFO:teuthology.orchestra.run.smithi066.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T01:46:02.102 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T01:46:02.102 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T01:46:02.102 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T01:46:02.103 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T01:46:02.103 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-26T01:46:02.103 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi066.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T01:46:02.103 DEBUG:teuthology.orchestra.run.smithi066:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T01:46:02.186 INFO:teuthology.orchestra.run.smithi066.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T01:46:02.186 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T01:46:02.186 DEBUG:teuthology.orchestra.run.smithi066:> sudo rpm -qi grub2-tools 2024-03-26T01:46:02.231 INFO:teuthology.orchestra.run.smithi066.stdout:Name : grub2-tools 2024-03-26T01:46:02.231 INFO:teuthology.orchestra.run.smithi066.stdout:Epoch : 1 2024-03-26T01:46:02.231 INFO:teuthology.orchestra.run.smithi066.stdout:Version : 2.06 2024-03-26T01:46:02.231 INFO:teuthology.orchestra.run.smithi066.stdout:Release : 68.el9 2024-03-26T01:46:02.231 INFO:teuthology.orchestra.run.smithi066.stdout:Architecture: x86_64 2024-03-26T01:46:02.231 INFO:teuthology.orchestra.run.smithi066.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T01:46:02.231 INFO:teuthology.orchestra.run.smithi066.stdout:Group : Unspecified 2024-03-26T01:46:02.232 INFO:teuthology.orchestra.run.smithi066.stdout:Size : 8264254 2024-03-26T01:46:02.232 INFO:teuthology.orchestra.run.smithi066.stdout:License : GPLv3+ 2024-03-26T01:46:02.232 INFO:teuthology.orchestra.run.smithi066.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T01:46:02.232 INFO:teuthology.orchestra.run.smithi066.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T01:46:02.232 INFO:teuthology.orchestra.run.smithi066.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T01:46:02.232 INFO:teuthology.orchestra.run.smithi066.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T01:46:02.232 INFO:teuthology.orchestra.run.smithi066.stdout:Packager : builder@centos.org 2024-03-26T01:46:02.232 INFO:teuthology.orchestra.run.smithi066.stdout:Vendor : CentOS 2024-03-26T01:46:02.232 INFO:teuthology.orchestra.run.smithi066.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T01:46:02.232 INFO:teuthology.orchestra.run.smithi066.stdout:Summary : Support tools for GRUB. 2024-03-26T01:46:02.232 INFO:teuthology.orchestra.run.smithi066.stdout:Description : 2024-03-26T01:46:02.232 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T01:46:02.232 INFO:teuthology.orchestra.run.smithi066.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T01:46:02.232 INFO:teuthology.orchestra.run.smithi066.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T01:46:02.233 INFO:teuthology.orchestra.run.smithi066.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T01:46:02.233 INFO:teuthology.orchestra.run.smithi066.stdout:hardware devices. 2024-03-26T01:46:02.233 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T01:46:02.233 INFO:teuthology.orchestra.run.smithi066.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T01:46:02.233 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T01:46:02.234 INFO:teuthology.task.kernel:Updating grub on smithi066 to boot 5.14.0-432.el9.x86_64 2024-03-26T01:46:02.234 DEBUG:teuthology.orchestra.run.smithi066:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T01:46:02.816 INFO:teuthology.orchestra.run.smithi066.stderr:Generating grub configuration file ... 2024-03-26T01:46:04.113 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:46:04.113 INFO:teuthology.orchestra.run.smithi003.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T01:46:04.113 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:46:04.159 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:46:04.186 INFO:teuthology.orchestra.run.smithi066.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T01:46:04.225 INFO:teuthology.orchestra.run.smithi066.stderr:done 2024-03-26T01:46:04.227 DEBUG:teuthology.orchestra.run.smithi066:> mktemp 2024-03-26T01:46:04.243 INFO:teuthology.orchestra.run.smithi066.stdout:/tmp/tmp.12Ymo1PE7S 2024-03-26T01:46:04.244 DEBUG:teuthology.orchestra.run.smithi066:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.12Ymo1PE7S 2024-03-26T01:46:04.315 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0666 /tmp/tmp.12Ymo1PE7S 2024-03-26T01:46:04.459 DEBUG:teuthology.orchestra.remote:smithi066:/tmp/tmp.12Ymo1PE7S is 6KB 2024-03-26T01:46:04.508 DEBUG:teuthology.orchestra.run.smithi066:> rm -fr /tmp/tmp.12Ymo1PE7S 2024-03-26T01:46:04.522 DEBUG:teuthology.orchestra.run.smithi066:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T01:46:04.549 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:46:04.550 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T01:46:04.550 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:46:04.550 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:46:04.590 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T01:46:04.591 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T01:46:04.591 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T01:46:04.591 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T01:46:04.592 DEBUG:teuthology.orchestra.run.smithi066:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T01:46:04.729 DEBUG:teuthology.orchestra.run.smithi066:> sudo shutdown -r now 2024-03-26T01:46:04.944 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:46:04.944 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:46:04.944 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-26T01:46:04.944 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T01:46:04.944 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T01:46:04.944 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T01:46:04.944 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T01:46:04.944 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:46:04.944 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-26T01:46:05.297 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T01:46:05.397 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T01:46:05.397 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T01:46:05.397 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T01:46:05.397 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T01:46:05.398 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T01:46:05.398 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-26T01:46:05.398 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi003.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T01:46:05.398 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T01:46:05.482 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T01:46:05.482 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T01:46:05.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo rpm -qi grub2-tools 2024-03-26T01:46:05.528 INFO:teuthology.orchestra.run.smithi003.stdout:Name : grub2-tools 2024-03-26T01:46:05.528 INFO:teuthology.orchestra.run.smithi003.stdout:Epoch : 1 2024-03-26T01:46:05.529 INFO:teuthology.orchestra.run.smithi003.stdout:Version : 2.06 2024-03-26T01:46:05.529 INFO:teuthology.orchestra.run.smithi003.stdout:Release : 68.el9 2024-03-26T01:46:05.529 INFO:teuthology.orchestra.run.smithi003.stdout:Architecture: x86_64 2024-03-26T01:46:05.529 INFO:teuthology.orchestra.run.smithi003.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T01:46:05.529 INFO:teuthology.orchestra.run.smithi003.stdout:Group : Unspecified 2024-03-26T01:46:05.529 INFO:teuthology.orchestra.run.smithi003.stdout:Size : 8264254 2024-03-26T01:46:05.529 INFO:teuthology.orchestra.run.smithi003.stdout:License : GPLv3+ 2024-03-26T01:46:05.529 INFO:teuthology.orchestra.run.smithi003.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T01:46:05.529 INFO:teuthology.orchestra.run.smithi003.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T01:46:05.529 INFO:teuthology.orchestra.run.smithi003.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T01:46:05.529 INFO:teuthology.orchestra.run.smithi003.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T01:46:05.529 INFO:teuthology.orchestra.run.smithi003.stdout:Packager : builder@centos.org 2024-03-26T01:46:05.530 INFO:teuthology.orchestra.run.smithi003.stdout:Vendor : CentOS 2024-03-26T01:46:05.530 INFO:teuthology.orchestra.run.smithi003.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T01:46:05.530 INFO:teuthology.orchestra.run.smithi003.stdout:Summary : Support tools for GRUB. 2024-03-26T01:46:05.530 INFO:teuthology.orchestra.run.smithi003.stdout:Description : 2024-03-26T01:46:05.530 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:46:05.530 INFO:teuthology.orchestra.run.smithi003.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T01:46:05.530 INFO:teuthology.orchestra.run.smithi003.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T01:46:05.530 INFO:teuthology.orchestra.run.smithi003.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T01:46:05.530 INFO:teuthology.orchestra.run.smithi003.stdout:hardware devices. 2024-03-26T01:46:05.530 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:46:05.530 INFO:teuthology.orchestra.run.smithi003.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T01:46:05.531 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T01:46:05.531 INFO:teuthology.task.kernel:Updating grub on smithi003 to boot 5.14.0-432.el9.x86_64 2024-03-26T01:46:05.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T01:46:06.126 INFO:teuthology.orchestra.run.smithi003.stderr:Generating grub configuration file ... 2024-03-26T01:46:07.508 INFO:teuthology.orchestra.run.smithi003.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T01:46:07.540 INFO:teuthology.orchestra.run.smithi003.stderr:done 2024-03-26T01:46:07.542 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-03-26T01:46:07.557 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.KTG3EUkTgS 2024-03-26T01:46:07.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KTG3EUkTgS 2024-03-26T01:46:07.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.KTG3EUkTgS 2024-03-26T01:46:07.778 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.KTG3EUkTgS is 6KB 2024-03-26T01:46:07.827 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.KTG3EUkTgS 2024-03-26T01:46:07.842 DEBUG:teuthology.orchestra.run.smithi003:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T01:46:07.920 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T01:46:07.920 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T01:46:07.920 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T01:46:07.920 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T01:46:07.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T01:46:08.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo shutdown -r now 2024-03-26T01:46:29.182 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-26T01:46:29.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-26T01:46:29.184 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:46:34.734 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-26T01:46:34.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-26T01:46:34.735 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:46:38.085 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-26T01:46:38.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-26T01:46:38.086 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:46:47.710 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.55 2024-03-26T01:46:53.090 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-03-26T01:46:56.715 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-26T01:46:56.716 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:47:02.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-26T01:47:02.097 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:47:05.150 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-03-26T01:47:17.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-26T01:47:17.160 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:47:20.222 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-03-26T01:47:28.858 DEBUG:teuthology.orchestra.run.smithi055:> true 2024-03-26T01:47:29.348 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-26T01:47:29.348 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-26T01:47:29.349 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2024-03-26T01:47:29.362 INFO:teuthology.orchestra.run.smithi055.stdout:5.14.0-432.el9.x86_64 2024-03-26T01:47:29.363 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T01:47:29.363 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T01:47:29.363 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T01:47:30.363 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-26T01:47:30.364 DEBUG:teuthology.orchestra.run.smithi055:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T01:47:30.454 INFO:teuthology.orchestra.run.smithi055.stdout:ttyS1 2024-03-26T01:47:30.510 DEBUG:teuthology.parallel:result is None 2024-03-26T01:47:35.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-26T01:47:35.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:47:38.089 DEBUG:teuthology.orchestra.remote:timed out 2024-03-26T01:47:38.306 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-03-26T01:47:47.090 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-26T01:47:47.091 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:47:47.453 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-03-26T01:47:48.030 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-26T01:47:48.030 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-26T01:47:48.030 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-03-26T01:47:48.060 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-432.el9.x86_64 2024-03-26T01:47:48.060 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T01:47:48.060 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T01:47:48.060 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T01:47:49.061 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-26T01:47:49.062 DEBUG:teuthology.orchestra.run.smithi003:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T01:47:49.126 INFO:teuthology.orchestra.run.smithi003.stdout:ttyS1 2024-03-26T01:47:49.167 DEBUG:teuthology.parallel:result is None 2024-03-26T01:47:56.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-26T01:47:56.308 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:48:56.310 DEBUG:teuthology.orchestra.remote:timed out 2024-03-26T01:49:17.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-26T01:49:17.313 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:49:17.652 DEBUG:teuthology.orchestra.run.smithi066:> true 2024-03-26T01:49:18.035 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-26T01:49:18.035 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-432.el9.x86_64"... 2024-03-26T01:49:18.035 DEBUG:teuthology.orchestra.run.smithi066:> uname -r 2024-03-26T01:49:18.051 INFO:teuthology.orchestra.run.smithi066.stdout:5.14.0-432.el9.x86_64 2024-03-26T01:49:18.051 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T01:49:18.051 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T01:49:18.051 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T01:49:19.052 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-26T01:49:19.052 DEBUG:teuthology.orchestra.run.smithi066:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T01:49:19.138 INFO:teuthology.orchestra.run.smithi066.stdout:ttyS1 2024-03-26T01:49:19.175 DEBUG:teuthology.parallel:result is None 2024-03-26T01:49:19.176 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-26T01:49:19.183 INFO:teuthology.task.internal:Creating test directory... 2024-03-26T01:49:19.184 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T01:49:19.186 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T01:49:19.190 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T01:49:19.207 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-26T01:49:19.214 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-26T01:49:19.222 INFO:teuthology.task.internal:Creating archive directory... 2024-03-26T01:49:19.222 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T01:49:19.245 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T01:49:19.248 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T01:49:19.280 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-26T01:49:19.287 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-26T01:49:19.287 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T01:49:19.313 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T01:49:19.315 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T01:49:19.349 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T01:49:19.350 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T01:49:19.354 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T01:49:19.364 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T01:49:19.364 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T01:49:19.371 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T01:49:19.373 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-26T01:49:19.380 INFO:teuthology.task.internal:Configuring sudo... 2024-03-26T01:49:19.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T01:49:19.409 DEBUG:teuthology.orchestra.run.smithi055:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T01:49:19.411 DEBUG:teuthology.orchestra.run.smithi066:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T01:49:19.446 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-26T01:49:19.456 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-26T01:49:19.457 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T01:49:19.479 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T01:49:19.481 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T01:49:19.502 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T01:49:19.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T01:49:19.635 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T01:49:19.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T01:49:19.725 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-26T01:49:19.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T01:49:19.792 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T01:49:19.836 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T01:49:19.895 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T01:49:19.979 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T01:49:20.023 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T01:49:20.024 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T01:49:20.092 DEBUG:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T01:49:20.137 DEBUG:teuthology.orchestra.run.smithi066:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T01:49:20.199 DEBUG:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T01:49:20.243 DEBUG:teuthology.orchestra.run.smithi066:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T01:49:20.292 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T01:49:20.292 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T01:49:20.366 DEBUG:teuthology.orchestra.run.smithi003:> sudo service rsyslog restart 2024-03-26T01:49:20.368 DEBUG:teuthology.orchestra.run.smithi055:> sudo service rsyslog restart 2024-03-26T01:49:20.370 DEBUG:teuthology.orchestra.run.smithi066:> sudo service rsyslog restart 2024-03-26T01:49:20.420 INFO:teuthology.orchestra.run.smithi055.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T01:49:20.421 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T01:49:20.453 INFO:teuthology.orchestra.run.smithi066.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T01:49:20.928 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-26T01:49:20.936 INFO:teuthology.task.internal:Starting timer... 2024-03-26T01:49:20.936 INFO:teuthology.run_tasks:Running task pcp... 2024-03-26T01:49:20.947 INFO:teuthology.run_tasks:Running task selinux... 2024-03-26T01:49:20.956 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-26T01:49:20.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo service auditd rotate 2024-03-26T01:49:21.009 INFO:teuthology.orchestra.run.smithi003.stdout:Rotating logs: 2024-03-26T01:49:21.011 DEBUG:teuthology.orchestra.run.smithi055:> sudo service auditd rotate 2024-03-26T01:49:21.080 INFO:teuthology.orchestra.run.smithi055.stdout:Rotating logs: 2024-03-26T01:49:21.082 DEBUG:teuthology.orchestra.run.smithi066:> sudo service auditd rotate 2024-03-26T01:49:21.129 INFO:teuthology.orchestra.run.smithi066.stdout:Rotating logs: 2024-03-26T01:49:21.131 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-26T01:49:21.131 DEBUG:teuthology.orchestra.run.smithi003:> /usr/sbin/getenforce 2024-03-26T01:49:21.158 INFO:teuthology.orchestra.run.smithi003.stdout:Permissive 2024-03-26T01:49:21.158 DEBUG:teuthology.orchestra.run.smithi055:> /usr/sbin/getenforce 2024-03-26T01:49:21.183 INFO:teuthology.orchestra.run.smithi055.stdout:Permissive 2024-03-26T01:49:21.183 DEBUG:teuthology.orchestra.run.smithi066:> /usr/sbin/getenforce 2024-03-26T01:49:21.203 INFO:teuthology.orchestra.run.smithi066.stdout:Permissive 2024-03-26T01:49:21.203 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi003.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi055.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi066.front.sepia.ceph.com': 'permissive'} 2024-03-26T01:49:21.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T01:49:21.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T01:49:21.234 DEBUG:teuthology.orchestra.run.smithi055:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T01:49:21.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T01:49:21.263 DEBUG:teuthology.orchestra.run.smithi066:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T01:49:21.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T01:49:21.296 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-26T01:49:21.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo /usr/sbin/setenforce permissive 2024-03-26T01:49:21.343 DEBUG:teuthology.orchestra.run.smithi055:> sudo /usr/sbin/setenforce permissive 2024-03-26T01:49:21.376 DEBUG:teuthology.orchestra.run.smithi066:> sudo /usr/sbin/setenforce permissive 2024-03-26T01:49:21.414 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-26T01:49:21.425 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-26T01:49:21.441 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-26T01:49:21.442 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi003.front.sepia.ceph.com,smithi055.front.sepia.ceph.com,smithi066.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-26T01:53:54.520 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi003.front.sepia.ceph.com'), Remote(name='ubuntu@smithi055.front.sepia.ceph.com'), Remote(name='ubuntu@smithi066.front.sepia.ceph.com')] 2024-03-26T01:53:54.521 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-26T01:53:54.523 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:53:54.615 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-03-26T01:53:54.688 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-26T01:53:54.689 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-26T01:53:54.689 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:53:54.779 DEBUG:teuthology.orchestra.run.smithi055:> true 2024-03-26T01:53:54.867 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-26T01:53:54.868 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-26T01:53:54.868 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:53:54.968 DEBUG:teuthology.orchestra.run.smithi066:> true 2024-03-26T01:53:55.044 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-26T01:53:55.045 INFO:teuthology.run_tasks:Running task clock... 2024-03-26T01:53:55.056 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-26T01:53:55.056 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T01:53:55.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T01:53:55.059 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T01:53:55.059 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T01:53:55.061 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T01:53:55.061 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T01:53:55.088 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T01:53:55.093 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T01:53:55.106 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T01:53:55.112 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T01:53:55.119 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T01:53:55.135 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: ntpd: command not found 2024-03-26T01:53:55.139 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T01:53:55.143 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: ntpd: command not found 2024-03-26T01:53:55.149 INFO:teuthology.orchestra.run.smithi003.stdout:506 Cannot talk to daemon 2024-03-26T01:53:55.158 INFO:teuthology.orchestra.run.smithi055.stdout:506 Cannot talk to daemon 2024-03-26T01:53:55.166 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T01:53:55.174 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: ntpd: command not found 2024-03-26T01:53:55.175 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T01:53:55.183 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T01:53:55.191 INFO:teuthology.orchestra.run.smithi066.stdout:506 Cannot talk to daemon 2024-03-26T01:53:55.192 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T01:53:55.211 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T01:53:55.231 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T01:53:55.239 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2024-03-26T01:53:55.243 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T01:53:55.243 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2024-03-26T01:53:55.243 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:53:55.243 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:53:55.243 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:53:55.243 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:53:55.255 INFO:teuthology.orchestra.run.smithi055.stderr:bash: line 1: ntpq: command not found 2024-03-26T01:53:55.259 INFO:teuthology.orchestra.run.smithi055.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T01:53:55.259 INFO:teuthology.orchestra.run.smithi055.stdout:=============================================================================== 2024-03-26T01:53:55.259 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:53:55.259 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:53:55.259 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:53:55.260 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:53:55.290 INFO:teuthology.orchestra.run.smithi066.stderr:bash: line 1: ntpq: command not found 2024-03-26T01:53:55.293 INFO:teuthology.orchestra.run.smithi066.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T01:53:55.293 INFO:teuthology.orchestra.run.smithi066.stdout:=============================================================================== 2024-03-26T01:53:55.293 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:53:55.293 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:53:55.293 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:53:55.294 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:53:55.294 INFO:teuthology.run_tasks:Running task pexec... 2024-03-26T01:53:55.307 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-26T01:53:55.307 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi003.front.sepia.ceph.com 2024-03-26T01:53:55.307 DEBUG:teuthology.orchestra.run.smithi003:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-26T01:53:55.308 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi055.front.sepia.ceph.com 2024-03-26T01:53:55.308 DEBUG:teuthology.orchestra.run.smithi055:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-26T01:53:55.308 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi066.front.sepia.ceph.com 2024-03-26T01:53:55.308 DEBUG:teuthology.orchestra.run.smithi066:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-26T01:53:55.900 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:03 ago on Tue 26 Mar 2024 01:52:52 AM UTC. 2024-03-26T01:53:55.900 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:03 ago on Tue 26 Mar 2024 01:52:52 AM UTC. 2024-03-26T01:53:55.942 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:00 ago on Tue 26 Mar 2024 01:52:55 AM UTC. 2024-03-26T01:53:56.051 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T01:53:56.051 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-26T01:53:56.060 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T01:53:56.060 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2024-03-26T01:53:56.060 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T01:53:56.060 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-26T01:53:56.060 INFO:teuthology.orchestra.run.smithi055.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-26T01:53:56.060 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T01:53:56.060 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-26T01:53:56.060 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T01:53:56.061 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-26T01:53:56.061 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T01:53:56.061 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 3.1 M 2024-03-26T01:53:56.061 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 9.8 M 2024-03-26T01:53:56.061 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-26T01:53:56.064 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-26T01:53:56.064 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2024-03-26T01:53:56.064 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-26T01:53:56.064 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-26T01:53:56.065 INFO:teuthology.orchestra.run.smithi003.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-26T01:53:56.065 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:53:56.065 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-26T01:53:56.065 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-26T01:53:56.065 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-26T01:53:56.065 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:53:56.065 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 3.1 M 2024-03-26T01:53:56.065 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 9.8 M 2024-03-26T01:53:56.066 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-26T01:53:56.095 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T01:53:56.097 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T01:53:56.097 INFO:teuthology.orchestra.run.smithi066.stdout: Package Architecture Version Repository Size 2024-03-26T01:53:56.097 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T01:53:56.097 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-26T01:53:56.098 INFO:teuthology.orchestra.run.smithi066.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-26T01:53:56.098 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T01:53:56.098 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-26T01:53:56.098 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T01:53:56.098 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-26T01:53:56.098 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T01:53:56.098 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 3.1 M 2024-03-26T01:53:56.098 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 9.8 M 2024-03-26T01:53:56.098 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-26T01:53:56.997 INFO:teuthology.orchestra.run.smithi066.stdout:runc-1.1.12-2.el9.x86_64.rpm 4.1 MB/s | 3.1 MB 00:00 2024-03-26T01:53:56.997 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:53:56.997 INFO:teuthology.orchestra.run.smithi066.stdout:Total 3.5 MB/s | 3.1 MB 00:00 2024-03-26T01:53:57.034 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-26T01:53:57.041 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-26T01:53:57.041 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-26T01:53:57.121 INFO:teuthology.orchestra.run.smithi003.stdout:runc-1.1.12-2.el9.x86_64.rpm 3.2 MB/s | 3.1 MB 00:00 2024-03-26T01:53:57.121 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:53:57.121 INFO:teuthology.orchestra.run.smithi003.stdout:Total 3.0 MB/s | 3.1 MB 00:01 2024-03-26T01:53:57.126 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-26T01:53:57.126 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-26T01:53:57.160 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-26T01:53:57.167 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-26T01:53:57.167 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-26T01:53:57.252 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-26T01:53:57.252 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-26T01:53:57.347 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-26T01:53:57.443 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T01:53:57.444 INFO:teuthology.orchestra.run.smithi055.stdout:runc-1.1.12-2.el9.x86_64.rpm 2.3 MB/s | 3.1 MB 00:01 2024-03-26T01:53:57.444 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:53:57.445 INFO:teuthology.orchestra.run.smithi055.stdout:Total 2.3 MB/s | 3.1 MB 00:01 2024-03-26T01:53:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-26T01:53:57.483 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-26T01:53:57.490 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-26T01:53:57.490 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-26T01:53:57.575 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-26T01:53:57.576 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-26T01:53:57.585 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T01:53:57.793 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-26T01:53:57.890 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T01:53:58.025 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T01:53:58.071 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T01:53:58.339 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T01:53:58.339 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:53:58.339 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-26T01:53:58.339 INFO:teuthology.orchestra.run.smithi003.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-26T01:53:58.339 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:53:58.340 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-26T01:53:58.390 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T01:53:58.390 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T01:53:58.390 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-26T01:53:58.390 INFO:teuthology.orchestra.run.smithi066.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-26T01:53:58.391 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T01:53:58.391 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T01:53:58.611 DEBUG:teuthology.parallel:result is None 2024-03-26T01:53:58.616 DEBUG:teuthology.parallel:result is None 2024-03-26T01:53:58.618 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T01:53:58.940 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T01:53:58.941 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T01:53:58.941 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-26T01:53:58.941 INFO:teuthology.orchestra.run.smithi055.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-26T01:53:58.941 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T01:53:58.941 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T01:53:59.172 DEBUG:teuthology.parallel:result is None 2024-03-26T01:53:59.172 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-26T01:53:59.184 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-26T01:53:59.185 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-26T01:53:59.185 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T01:53:59.202 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T01:53:59.202 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_1 2024-03-26T01:53:59.259 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T01:53:59.259 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:53:59.259 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-26T01:53:59.259 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:53:59.259 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:53:59.259 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-26 01:53:53.453845461 +0000 2024-03-26T01:53:59.259 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-26 01:53:17.840284202 +0000 2024-03-26T01:53:59.260 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-26 01:53:17.840284202 +0000 2024-03-26T01:53:59.260 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-26 01:53:17.840284202 +0000 2024-03-26T01:53:59.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T01:53:59.328 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-26T01:53:59.328 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-26T01:53:59.328 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000194982 s, 2.6 MB/s 2024-03-26T01:53:59.329 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T01:53:59.387 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_2 2024-03-26T01:53:59.443 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T01:53:59.443 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:53:59.443 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-26T01:53:59.443 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:53:59.443 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:53:59.444 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-26 01:53:53.453845461 +0000 2024-03-26T01:53:59.444 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-26 01:53:18.170280137 +0000 2024-03-26T01:53:59.444 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-26 01:53:18.170280137 +0000 2024-03-26T01:53:59.444 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-26 01:53:18.170280137 +0000 2024-03-26T01:53:59.444 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T01:53:59.510 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-26T01:53:59.510 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-26T01:53:59.510 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000182494 s, 2.8 MB/s 2024-03-26T01:53:59.511 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T01:53:59.568 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_3 2024-03-26T01:53:59.623 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T01:53:59.624 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:53:59.624 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-26T01:53:59.624 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:53:59.624 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:53:59.624 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-26 01:53:53.453845461 +0000 2024-03-26T01:53:59.624 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-26 01:53:18.531275690 +0000 2024-03-26T01:53:59.624 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-26 01:53:18.531275690 +0000 2024-03-26T01:53:59.624 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-26 01:53:18.531275690 +0000 2024-03-26T01:53:59.624 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T01:53:59.690 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-26T01:53:59.690 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-26T01:53:59.690 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000188124 s, 2.7 MB/s 2024-03-26T01:53:59.691 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T01:53:59.748 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_4 2024-03-26T01:53:59.804 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T01:53:59.804 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:53:59.804 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-26T01:53:59.804 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:53:59.804 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:53:59.804 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-26 01:53:53.453845461 +0000 2024-03-26T01:53:59.804 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-26 01:53:18.870271513 +0000 2024-03-26T01:53:59.804 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-26 01:53:18.870271513 +0000 2024-03-26T01:53:59.804 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-26 01:53:18.870271513 +0000 2024-03-26T01:53:59.805 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T01:53:59.870 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-26T01:53:59.870 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-26T01:53:59.871 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000218029 s, 2.3 MB/s 2024-03-26T01:53:59.872 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T01:53:59.928 DEBUG:teuthology.orchestra.run.smithi003:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-26T01:54:00.068 INFO:teuthology.orchestra.run.smithi003.stdout:loop 2024-03-26T01:54:00.069 INFO:tasks.nvme_loop:Connecting nvme_loop smithi003:/dev/vg_nvme/lv_1... 2024-03-26T01:54:00.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-26T01:54:00.107 INFO:teuthology.orchestra.run.smithi003.stdout:1 2024-03-26T01:54:00.144 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/vg_nvme/lv_11 2024-03-26T01:54:00.174 INFO:tasks.nvme_loop:Connecting nvme_loop smithi003:/dev/vg_nvme/lv_2... 2024-03-26T01:54:00.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-26T01:54:00.252 INFO:teuthology.orchestra.run.smithi003.stdout:1 2024-03-26T01:54:00.294 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/vg_nvme/lv_21 2024-03-26T01:54:00.331 INFO:tasks.nvme_loop:Connecting nvme_loop smithi003:/dev/vg_nvme/lv_3... 2024-03-26T01:54:00.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-26T01:54:00.411 INFO:teuthology.orchestra.run.smithi003.stdout:1 2024-03-26T01:54:00.450 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/vg_nvme/lv_31 2024-03-26T01:54:00.486 INFO:tasks.nvme_loop:Connecting nvme_loop smithi003:/dev/vg_nvme/lv_4... 2024-03-26T01:54:00.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-26T01:54:00.568 INFO:teuthology.orchestra.run.smithi003.stdout:1 2024-03-26T01:54:00.607 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/vg_nvme/lv_41 2024-03-26T01:54:00.638 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-26T01:54:00.638 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T01:54:00.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo nvme list 2024-03-26T01:54:00.764 INFO:teuthology.orchestra.run.smithi003.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-26T01:54:00.764 INFO:teuthology.orchestra.run.smithi003.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-26T01:54:00.765 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/nvme0n1 CVFT5330009Z400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-26T01:54:00.765 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/nvme1n1 0a0d385bc5bfdb616bfc Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T01:54:00.765 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/nvme2n1 74094c002ec1a6356a99 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T01:54:00.765 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/nvme3n1 63e9e6c6c5772d9eb7c8 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T01:54:00.765 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/nvme4n1 289d483ded2146e17ecb Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T01:54:00.766 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T01:54:00.766 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-26T01:54:00.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/scratch_devs 2024-03-26T01:54:00.833 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T01:54:00.833 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T01:54:00.851 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T01:54:00.851 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_1 2024-03-26T01:54:00.909 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T01:54:00.909 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:54:00.909 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-26T01:54:00.909 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:54:00.909 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:54:00.909 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-26 01:53:53.520395146 +0000 2024-03-26T01:54:00.910 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-26 01:53:18.095122676 +0000 2024-03-26T01:54:00.910 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-26 01:53:18.095122676 +0000 2024-03-26T01:54:00.910 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-03-26 01:53:18.095122676 +0000 2024-03-26T01:54:00.910 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T01:54:00.978 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-26T01:54:00.979 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-26T01:54:00.979 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000289915 s, 1.8 MB/s 2024-03-26T01:54:00.980 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T01:54:01.039 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_2 2024-03-26T01:54:01.096 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T01:54:01.096 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:54:01.096 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-26T01:54:01.096 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:54:01.096 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:54:01.096 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-26 01:53:53.520395146 +0000 2024-03-26T01:54:01.096 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-26 01:53:18.452115345 +0000 2024-03-26T01:54:01.096 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-26 01:53:18.452115345 +0000 2024-03-26T01:54:01.096 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-03-26 01:53:18.452115345 +0000 2024-03-26T01:54:01.097 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T01:54:01.165 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-26T01:54:01.165 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-26T01:54:01.165 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000300556 s, 1.7 MB/s 2024-03-26T01:54:01.167 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T01:54:01.224 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_3 2024-03-26T01:54:01.281 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T01:54:01.281 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:54:01.281 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-26T01:54:01.281 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:54:01.281 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:54:01.281 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-26 01:53:53.520395146 +0000 2024-03-26T01:54:01.281 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-26 01:53:18.795108301 +0000 2024-03-26T01:54:01.281 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-26 01:53:18.795108301 +0000 2024-03-26T01:54:01.281 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-03-26 01:53:18.795108301 +0000 2024-03-26T01:54:01.282 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T01:54:01.347 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-26T01:54:01.347 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-26T01:54:01.347 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000264579 s, 1.9 MB/s 2024-03-26T01:54:01.348 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T01:54:01.404 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_4 2024-03-26T01:54:01.460 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T01:54:01.461 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:54:01.461 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-26T01:54:01.461 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:54:01.461 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:54:01.461 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-26 01:53:53.520395146 +0000 2024-03-26T01:54:01.461 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-26 01:53:19.144101133 +0000 2024-03-26T01:54:01.461 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-26 01:53:19.144101133 +0000 2024-03-26T01:54:01.462 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-03-26 01:53:19.144101133 +0000 2024-03-26T01:54:01.462 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T01:54:01.526 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-26T01:54:01.526 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-26T01:54:01.526 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000248367 s, 2.1 MB/s 2024-03-26T01:54:01.527 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T01:54:01.582 DEBUG:teuthology.orchestra.run.smithi055:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-26T01:54:01.724 INFO:teuthology.orchestra.run.smithi055.stdout:loop 2024-03-26T01:54:01.725 INFO:tasks.nvme_loop:Connecting nvme_loop smithi055:/dev/vg_nvme/lv_1... 2024-03-26T01:54:01.725 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-26T01:54:01.762 INFO:teuthology.orchestra.run.smithi055.stdout:1 2024-03-26T01:54:01.800 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/vg_nvme/lv_11 2024-03-26T01:54:01.830 INFO:tasks.nvme_loop:Connecting nvme_loop smithi055:/dev/vg_nvme/lv_2... 2024-03-26T01:54:01.830 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-26T01:54:01.907 INFO:teuthology.orchestra.run.smithi055.stdout:1 2024-03-26T01:54:01.945 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/vg_nvme/lv_21 2024-03-26T01:54:01.973 INFO:tasks.nvme_loop:Connecting nvme_loop smithi055:/dev/vg_nvme/lv_3... 2024-03-26T01:54:01.973 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-26T01:54:02.056 INFO:teuthology.orchestra.run.smithi055.stdout:1 2024-03-26T01:54:02.096 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/vg_nvme/lv_31 2024-03-26T01:54:02.125 INFO:tasks.nvme_loop:Connecting nvme_loop smithi055:/dev/vg_nvme/lv_4... 2024-03-26T01:54:02.125 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-26T01:54:02.164 INFO:teuthology.orchestra.run.smithi055.stdout:1 2024-03-26T01:54:02.202 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/vg_nvme/lv_41 2024-03-26T01:54:02.230 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T01:54:02.231 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T01:54:02.285 DEBUG:teuthology.orchestra.run.smithi055:> sudo nvme list 2024-03-26T01:54:02.351 INFO:teuthology.orchestra.run.smithi055.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-26T01:54:02.351 INFO:teuthology.orchestra.run.smithi055.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-26T01:54:02.351 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/nvme0n1 BTLE736102VS4P0KGN INTEL SSDPEDKE040T7 1 4.00 TB / 4.00 TB 512 B + 0 B QDV1013D 2024-03-26T01:54:02.352 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/nvme1n1 946394b4ad2caf1bfbcc Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T01:54:02.352 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/nvme2n1 56050468d35d5895d437 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T01:54:02.352 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/nvme3n1 536c0db306eadca59576 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T01:54:02.352 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/nvme4n1 5ec00d0f00f1bfc54eba Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T01:54:02.353 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T01:54:02.353 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T01:54:02.353 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/scratch_devs 2024-03-26T01:54:02.420 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T01:54:02.420 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T01:54:02.440 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T01:54:02.440 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_1 2024-03-26T01:54:02.499 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T01:54:02.499 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:54:02.499 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-26T01:54:02.499 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:54:02.499 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:54:02.499 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-26 01:53:53.544172998 +0000 2024-03-26T01:54:02.499 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-26 01:53:21.188458654 +0000 2024-03-26T01:54:02.499 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-26 01:53:21.188458654 +0000 2024-03-26T01:54:02.500 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-26 01:53:21.188458654 +0000 2024-03-26T01:54:02.500 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T01:54:02.574 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-26T01:54:02.574 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-26T01:54:02.574 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000254943 s, 2.0 MB/s 2024-03-26T01:54:02.576 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T01:54:02.635 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_2 2024-03-26T01:54:02.694 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T01:54:02.694 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:54:02.694 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-26T01:54:02.694 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:54:02.694 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:54:02.694 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-26 01:53:53.544172998 +0000 2024-03-26T01:54:02.694 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-26 01:53:21.558455388 +0000 2024-03-26T01:54:02.694 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-26 01:53:21.558455388 +0000 2024-03-26T01:54:02.695 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-26 01:53:21.558455388 +0000 2024-03-26T01:54:02.695 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T01:54:02.767 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-26T01:54:02.767 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-26T01:54:02.767 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000242927 s, 2.1 MB/s 2024-03-26T01:54:02.768 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T01:54:02.825 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_3 2024-03-26T01:54:02.881 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T01:54:02.882 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:54:02.882 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-26T01:54:02.882 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:54:02.882 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:54:02.882 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-26 01:53:53.544172998 +0000 2024-03-26T01:54:02.882 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-26 01:53:21.914452244 +0000 2024-03-26T01:54:02.882 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-26 01:53:21.914452244 +0000 2024-03-26T01:54:02.882 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-26 01:53:21.914452244 +0000 2024-03-26T01:54:02.882 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T01:54:02.953 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-26T01:54:03.009 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-26T01:54:03.009 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000189237 s, 2.7 MB/s 2024-03-26T01:54:03.010 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T01:54:03.068 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_4 2024-03-26T01:54:03.124 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T01:54:03.124 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:54:03.124 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-26T01:54:03.124 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:54:03.125 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:54:03.125 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-26 01:53:53.544172998 +0000 2024-03-26T01:54:03.125 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-26 01:53:22.271449093 +0000 2024-03-26T01:54:03.125 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-26 01:53:22.271449093 +0000 2024-03-26T01:54:03.125 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-26 01:53:22.271449093 +0000 2024-03-26T01:54:03.125 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T01:54:03.194 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-26T01:54:03.194 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-26T01:54:03.194 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000175683 s, 2.9 MB/s 2024-03-26T01:54:03.195 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T01:54:03.251 DEBUG:teuthology.orchestra.run.smithi066:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-26T01:54:03.412 INFO:teuthology.orchestra.run.smithi066.stdout:loop 2024-03-26T01:54:03.413 INFO:tasks.nvme_loop:Connecting nvme_loop smithi066:/dev/vg_nvme/lv_1... 2024-03-26T01:54:03.413 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-26T01:54:03.458 INFO:teuthology.orchestra.run.smithi066.stdout:1 2024-03-26T01:54:03.506 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/vg_nvme/lv_11 2024-03-26T01:54:03.543 INFO:tasks.nvme_loop:Connecting nvme_loop smithi066:/dev/vg_nvme/lv_2... 2024-03-26T01:54:03.543 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-26T01:54:03.590 INFO:teuthology.orchestra.run.smithi066.stdout:1 2024-03-26T01:54:03.639 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/vg_nvme/lv_21 2024-03-26T01:54:03.674 INFO:tasks.nvme_loop:Connecting nvme_loop smithi066:/dev/vg_nvme/lv_3... 2024-03-26T01:54:03.674 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-26T01:54:03.719 INFO:teuthology.orchestra.run.smithi066.stdout:1 2024-03-26T01:54:03.767 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/vg_nvme/lv_31 2024-03-26T01:54:03.802 INFO:tasks.nvme_loop:Connecting nvme_loop smithi066:/dev/vg_nvme/lv_4... 2024-03-26T01:54:03.803 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-26T01:54:03.860 INFO:teuthology.orchestra.run.smithi066.stdout:1 2024-03-26T01:54:03.908 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/vg_nvme/lv_41 2024-03-26T01:54:03.943 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T01:54:03.943 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T01:54:03.958 DEBUG:teuthology.orchestra.run.smithi066:> sudo nvme list 2024-03-26T01:54:04.027 INFO:teuthology.orchestra.run.smithi066.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-26T01:54:04.027 INFO:teuthology.orchestra.run.smithi066.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-26T01:54:04.027 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/nvme0n1 CVFT53430091400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-26T01:54:04.027 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/nvme1n1 8a4618dc01cba413d408 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T01:54:04.027 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/nvme2n1 bb1711b6ce20a44bf57c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T01:54:04.027 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/nvme3n1 56c7c153a120656585f3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T01:54:04.028 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/nvme4n1 6f879da8a4139b8483be Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T01:54:04.028 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T01:54:04.029 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T01:54:04.029 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/scratch_devs 2024-03-26T01:54:04.100 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-26T01:54:04.195 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'global': {'mon election default strategy': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a'} 2024-03-26T01:54:04.195 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:54:04.196 INFO:tasks.cephadm:Cluster fsid is b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T01:54:04.196 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-26T01:54:04.196 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.3', 'mon.b': '172.21.15.55', 'mon.c': '172.21.15.66'} 2024-03-26T01:54:04.196 INFO:tasks.cephadm:First mon is mon.a on smithi003 2024-03-26T01:54:04.196 INFO:tasks.cephadm:First mgr is y 2024-03-26T01:54:04.196 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-26T01:54:04.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo hostname $(hostname -s) 2024-03-26T01:54:04.229 DEBUG:teuthology.orchestra.run.smithi055:> sudo hostname $(hostname -s) 2024-03-26T01:54:04.262 DEBUG:teuthology.orchestra.run.smithi066:> sudo hostname $(hostname -s) 2024-03-26T01:54:04.293 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-26T01:54:04.293 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:54:04.445 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54690/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1775-g5e70124b', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1775.g5e70124b'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-25 18:23:16.641218', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-26T01:54:04.813 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref squid, sha1 5e70124b46c2a671efcaae4690cd9ea72670710a from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:54:04.815 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm 2024-03-26T01:54:04.815 DEBUG:teuthology.orchestra.run.smithi003:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T01:54:06.501 INFO:teuthology.orchestra.run.smithi003.stdout:-rw-r--r--. 1 ubuntu ubuntu 766713 Mar 26 01:54 /home/ubuntu/cephtest/cephadm 2024-03-26T01:54:06.501 DEBUG:teuthology.orchestra.run.smithi055:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T01:54:07.150 INFO:teuthology.orchestra.run.smithi055.stdout:-rw-r--r--. 1 ubuntu ubuntu 766713 Mar 26 01:54 /home/ubuntu/cephtest/cephadm 2024-03-26T01:54:07.151 DEBUG:teuthology.orchestra.run.smithi066:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T01:54:07.801 INFO:teuthology.orchestra.run.smithi066.stdout:-rw-r--r--. 1 ubuntu ubuntu 766713 Mar 26 01:54 /home/ubuntu/cephtest/cephadm 2024-03-26T01:54:07.801 DEBUG:teuthology.orchestra.run.smithi003:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T01:54:07.820 DEBUG:teuthology.orchestra.run.smithi055:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T01:54:07.838 DEBUG:teuthology.orchestra.run.smithi066:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T01:54:07.870 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a on all hosts... 2024-03-26T01:54:07.870 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T01:54:07.872 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T01:54:07.881 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T01:54:08.741 INFO:teuthology.orchestra.run.smithi003.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T01:54:08.749 INFO:teuthology.orchestra.run.smithi066.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T01:54:08.767 INFO:teuthology.orchestra.run.smithi055.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T01:54:36.548 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T01:54:36.548 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T01:54:36.549 INFO:teuthology.orchestra.run.smithi055.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T01:54:36.549 INFO:teuthology.orchestra.run.smithi055.stdout: "repo_digests": [ 2024-03-26T01:54:36.549 INFO:teuthology.orchestra.run.smithi055.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T01:54:36.549 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T01:54:36.549 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T01:54:36.732 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-26T01:54:36.732 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T01:54:36.733 INFO:teuthology.orchestra.run.smithi003.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T01:54:36.733 INFO:teuthology.orchestra.run.smithi003.stdout: "repo_digests": [ 2024-03-26T01:54:36.733 INFO:teuthology.orchestra.run.smithi003.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T01:54:36.733 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-26T01:54:36.733 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-26T01:54:38.257 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T01:54:38.257 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T01:54:38.257 INFO:teuthology.orchestra.run.smithi066.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T01:54:38.257 INFO:teuthology.orchestra.run.smithi066.stdout: "repo_digests": [ 2024-03-26T01:54:38.257 INFO:teuthology.orchestra.run.smithi066.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T01:54:38.257 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T01:54:38.258 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T01:54:38.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph 2024-03-26T01:54:38.318 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /etc/ceph 2024-03-26T01:54:38.354 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /etc/ceph 2024-03-26T01:54:38.388 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 777 /etc/ceph 2024-03-26T01:54:38.418 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 777 /etc/ceph 2024-03-26T01:54:38.447 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 777 /etc/ceph 2024-03-26T01:54:38.478 INFO:tasks.cephadm:Writing seed config... 2024-03-26T01:54:38.480 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-26T01:54:38.480 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-26T01:54:38.480 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-03-26T01:54:38.480 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-26T01:54:38.480 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-26T01:54:38.480 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-26T01:54:38.480 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-26T01:54:38.480 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-26T01:54:38.480 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-26T01:54:38.480 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-26T01:54:38.481 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-26T01:54:38.481 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-26T01:54:38.497 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = b95eae18-eb13-11ee-95d0-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-26T01:54:38.497 DEBUG:teuthology.orchestra.run.smithi003:mon.a> sudo journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mon.a.service 2024-03-26T01:54:38.539 DEBUG:teuthology.orchestra.run.smithi003:mgr.y> sudo journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mgr.y.service 2024-03-26T01:54:38.582 INFO:tasks.cephadm:Bootstrapping... 2024-03-26T01:54:38.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a -v bootstrap --fsid b95eae18-eb13-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.3 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-26T01:54:38.781 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:54:38.781 INFO:teuthology.orchestra.run.smithi003.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a', '-v', 'bootstrap', '--fsid', 'b95eae18-eb13-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.3', '--skip-admin-label'] 2024-03-26T01:54:38.805 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T01:54:38.806 INFO:teuthology.orchestra.run.smithi003.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-26T01:54:38.806 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying podman|docker is present... 2024-03-26T01:54:38.828 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T01:54:38.829 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying lvm2 is present... 2024-03-26T01:54:38.829 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying time synchronization is in place... 2024-03-26T01:54:38.836 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T01:54:38.836 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T01:54:38.841 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T01:54:38.841 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-03-26T01:54:38.847 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout enabled 2024-03-26T01:54:38.853 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout active 2024-03-26T01:54:38.853 INFO:teuthology.orchestra.run.smithi003.stdout:Unit chronyd.service is enabled and running 2024-03-26T01:54:38.853 INFO:teuthology.orchestra.run.smithi003.stdout:Repeating the final host check... 2024-03-26T01:54:38.874 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T01:54:38.875 INFO:teuthology.orchestra.run.smithi003.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-26T01:54:38.875 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl is present 2024-03-26T01:54:38.875 INFO:teuthology.orchestra.run.smithi003.stdout:lvcreate is present 2024-03-26T01:54:38.881 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T01:54:38.881 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T01:54:38.886 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T01:54:38.886 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-03-26T01:54:38.892 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout enabled 2024-03-26T01:54:38.903 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout active 2024-03-26T01:54:38.903 INFO:teuthology.orchestra.run.smithi003.stdout:Unit chronyd.service is enabled and running 2024-03-26T01:54:38.903 INFO:teuthology.orchestra.run.smithi003.stdout:Host looks OK 2024-03-26T01:54:38.903 INFO:teuthology.orchestra.run.smithi003.stdout:Cluster fsid: b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T01:54:38.903 INFO:teuthology.orchestra.run.smithi003.stdout:Acquiring lock 139899279915616 on /run/cephadm/b95eae18-eb13-11ee-95d0-87774f69a715.lock 2024-03-26T01:54:38.903 INFO:teuthology.orchestra.run.smithi003.stdout:Lock 139899279915616 acquired on /run/cephadm/b95eae18-eb13-11ee-95d0-87774f69a715.lock 2024-03-26T01:54:38.903 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying IP 172.21.15.3 port 3300 ... 2024-03-26T01:54:38.903 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying IP 172.21.15.3 port 6789 ... 2024-03-26T01:54:38.903 INFO:teuthology.orchestra.run.smithi003.stdout:Base mon IP(s) is [172.21.15.3:3300, 172.21.15.3:6789], mon addrv is [v2:172.21.15.3:3300,v1:172.21.15.3:6789] 2024-03-26T01:54:38.904 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.3 metric 100 2024-03-26T01:54:38.904 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.3 metric 100 2024-03-26T01:54:38.905 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-26T01:54:38.905 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-26T01:54:38.905 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-26T01:54:38.907 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-26T01:54:38.907 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-26T01:54:38.907 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T01:54:38.907 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-26T01:54:38.908 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15dc/64 scope link noprefixroute 2024-03-26T01:54:38.908 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T01:54:38.908 INFO:teuthology.orchestra.run.smithi003.stdout:Mon IP `172.21.15.3` is in CIDR network `172.21.0.0/20` 2024-03-26T01:54:38.908 INFO:teuthology.orchestra.run.smithi003.stdout:Mon IP `172.21.15.3` is in CIDR network `172.21.0.0/20` 2024-03-26T01:54:38.909 INFO:teuthology.orchestra.run.smithi003.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-26T01:54:38.909 INFO:teuthology.orchestra.run.smithi003.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-26T01:54:38.909 INFO:teuthology.orchestra.run.smithi003.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T01:54:39.368 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T01:54:39.368 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T01:54:39.368 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Getting image source signatures 2024-03-26T01:54:39.368 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Copying blob sha256:45ddbf13772d0d6535ecfdcaa58ecfc496e5849512dfbf602c5218332a216bcc 2024-03-26T01:54:39.368 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-26T01:54:39.368 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Copying config sha256:5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T01:54:39.368 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-26T01:54:40.373 INFO:teuthology.orchestra.run.smithi003.stdout:ceph: stdout ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev) 2024-03-26T01:54:40.373 INFO:teuthology.orchestra.run.smithi003.stdout:Ceph version: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev) 2024-03-26T01:54:40.373 INFO:teuthology.orchestra.run.smithi003.stdout:Extracting ceph user uid/gid from container image... 2024-03-26T01:54:41.278 INFO:teuthology.orchestra.run.smithi003.stdout:stat: stdout 167 167 2024-03-26T01:54:41.278 INFO:teuthology.orchestra.run.smithi003.stdout:Creating initial keys... 2024-03-26T01:54:42.307 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph-authtool: stdout AQDhKgJmvQmfJRAAbnvMSBwJBREvYpgKscbVnQ== 2024-03-26T01:54:43.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph-authtool: stdout AQDiKgJmoEB1JxAA2C+bSopk05hRLuvPoKyPfA== 2024-03-26T01:54:44.218 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph-authtool: stdout AQDjKgJmAbrDIRAAPy2/BTwvq5YanZgo5mItow== 2024-03-26T01:54:44.218 INFO:teuthology.orchestra.run.smithi003.stdout:Creating initial monmap... 2024-03-26T01:54:45.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T01:54:45.157 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-26T01:54:45.157 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T01:54:45.157 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T01:54:45.157 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool for a [v2:172.21.15.3:3300,v1:172.21.15.3:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T01:54:45.157 INFO:teuthology.orchestra.run.smithi003.stdout:setting min_mon_release = quincy 2024-03-26T01:54:45.157 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: set fsid to b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T01:54:45.157 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T01:54:45.157 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:54:45.157 INFO:teuthology.orchestra.run.smithi003.stdout:Creating mon... 2024-03-26T01:54:46.120 INFO:teuthology.orchestra.run.smithi003.stdout:create mon.a on 2024-03-26T01:54:46.470 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-26T01:54:46.643 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-b95eae18-eb13-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-b95eae18-eb13-11ee-95d0-87774f69a715.target. 2024-03-26T01:54:46.643 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-b95eae18-eb13-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-b95eae18-eb13-11ee-95d0-87774f69a715.target. 2024-03-26T01:54:46.893 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mon.a 2024-03-26T01:54:46.893 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to reset failed state of unit ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-26T01:54:47.052 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-b95eae18-eb13-11ee-95d0-87774f69a715.target.wants/ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-b95eae18-eb13-11ee-95d0-87774f69a715@.service. 2024-03-26T01:54:47.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:47 smithi003 podman[25287]: 2024-03-26 01:54:47.294259811 +0000 UTC m=+0.115445536 container create 3697b9f8e41141971e91b9f6bef104e4958981715d97cb34a9d93ceb0b69a78a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True) 2024-03-26T01:54:47.609 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T01:54:47.609 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2024-03-26T01:54:47.616 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T01:54:47.616 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-03-26T01:54:47.616 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2024-03-26T01:54:47.616 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T01:54:47.616 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mon to start... 2024-03-26T01:54:47.616 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mon... 2024-03-26T01:54:47.738 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:47 smithi003 podman[25287]: 2024-03-26 01:54:47.479725816 +0000 UTC m=+0.300911542 container init 3697b9f8e41141971e91b9f6bef104e4958981715d97cb34a9d93ceb0b69a78a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-a, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-03-26T01:54:47.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:47 smithi003 podman[25287]: 2024-03-26 01:54:47.485055992 +0000 UTC m=+0.306241719 container start 3697b9f8e41141971e91b9f6bef104e4958981715d97cb34a9d93ceb0b69a78a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-26T01:54:47.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:47 smithi003 ceph-mon[25325]: mkfs b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T01:54:47.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:47 smithi003 ceph-mon[25325]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T01:54:47.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:47 smithi003 bash[25287]: 3697b9f8e41141971e91b9f6bef104e4958981715d97cb34a9d93ceb0b69a78a 2024-03-26T01:54:47.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:47 smithi003 systemd[1]: Started Ceph mon.a for b95eae18-eb13-11ee-95d0-87774f69a715. 2024-03-26T01:54:48.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout cluster: 2024-03-26T01:54:48.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout id: b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T01:54:48.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-26T01:54:48.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-26T01:54:48.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout services: 2024-03-26T01:54:48.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.58472s) 2024-03-26T01:54:48.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-26T01:54:48.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-26T01:54:48.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-26T01:54:48.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout data: 2024-03-26T01:54:48.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-26T01:54:48.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-26T01:54:48.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-26T01:54:48.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout pgs: 2024-03-26T01:54:48.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-26T01:54:48.682 INFO:teuthology.orchestra.run.smithi003.stdout:mon is available 2024-03-26T01:54:48.682 INFO:teuthology.orchestra.run.smithi003.stdout:Assimilating anything we can from ceph.conf... 2024-03-26T01:54:48.794 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:48 smithi003 ceph-mon[25325]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T01:54:48.794 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:48 smithi003 ceph-mon[25325]: monmap e1: 1 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:54:48.794 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:48 smithi003 ceph-mon[25325]: fsmap 2024-03-26T01:54:48.794 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:48 smithi003 ceph-mon[25325]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T01:54:48.794 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:48 smithi003 ceph-mon[25325]: mgrmap e1: no daemons active 2024-03-26T01:54:48.794 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:48 smithi003 ceph-mon[25325]: from='client.? 172.21.15.3:0/2688691325' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T01:54:49.741 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:49 smithi003 ceph-mon[25325]: from='client.? 172.21.15.3:0/781696285' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T01:54:49.742 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:49 smithi003 ceph-mon[25325]: from='client.? 172.21.15.3:0/781696285' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T01:54:49.746 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-26T01:54:49.746 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T01:54:49.746 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout fsid = b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T01:54:49.747 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.3:3300,v1:172.21.15.3:6789] 2024-03-26T01:54:49.747 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T01:54:49.747 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T01:54:49.747 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T01:54:49.747 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T01:54:49.747 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-26T01:54:49.747 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T01:54:49.747 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = False 2024-03-26T01:54:49.747 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T01:54:49.747 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-26T01:54:49.747 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T01:54:49.747 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T01:54:49.748 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T01:54:49.748 INFO:teuthology.orchestra.run.smithi003.stdout:Generating new minimal ceph.conf... 2024-03-26T01:54:50.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:50 smithi003 ceph-mon[25325]: from='client.? 172.21.15.3:0/2490654279' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:54:50.815 INFO:teuthology.orchestra.run.smithi003.stdout:Restarting the monitor... 2024-03-26T01:54:51.111 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:50 smithi003 systemd[1]: Stopping Ceph mon.a for b95eae18-eb13-11ee-95d0-87774f69a715... 2024-03-26T01:54:51.111 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:51 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-a[25302]: 2024-03-26T01:54:51.059+0000 7f36536bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T01:54:51.111 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:51 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-a[25302]: 2024-03-26T01:54:51.059+0000 7f36536bb700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-26T01:54:51.366 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:51 smithi003 podman[25656]: 2024-03-26 01:54:51.111867546 +0000 UTC m=+0.135521666 container died 3697b9f8e41141971e91b9f6bef104e4958981715d97cb34a9d93ceb0b69a78a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True) 2024-03-26T01:54:51.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:51 smithi003 podman[25656]: 2024-03-26 01:54:51.463603787 +0000 UTC m=+0.487257948 container cleanup 3697b9f8e41141971e91b9f6bef104e4958981715d97cb34a9d93ceb0b69a78a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-a, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-26T01:54:51.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:51 smithi003 bash[25656]: ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-a 2024-03-26T01:54:52.092 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:51 smithi003 podman[25675]: 2024-03-26 01:54:51.733516482 +0000 UTC m=+0.623401760 container remove 3697b9f8e41141971e91b9f6bef104e4958981715d97cb34a9d93ceb0b69a78a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-a, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T01:54:52.092 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:51 smithi003 systemd[1]: ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-26T01:54:52.092 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:51 smithi003 systemd[1]: Stopped Ceph mon.a for b95eae18-eb13-11ee-95d0-87774f69a715. 2024-03-26T01:54:52.092 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:51 smithi003 systemd[1]: Starting Ceph mon.a for b95eae18-eb13-11ee-95d0-87774f69a715... 2024-03-26T01:54:52.344 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 podman[25772]: 2024-03-26 01:54:52.092641825 +0000 UTC m=+0.103749107 container create 8ed421558ef9f3e7ae4c71a89a2dc723a4bd61983231292a8684760900ce059d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T01:54:52.344 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 podman[25772]: 2024-03-26 01:54:52.006484663 +0000 UTC m=+0.017591947 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:54:52.344 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 podman[25772]: 2024-03-26 01:54:52.269633282 +0000 UTC m=+0.280740566 container init 8ed421558ef9f3e7ae4c71a89a2dc723a4bd61983231292a8684760900ce059d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-26T01:54:52.344 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 podman[25772]: 2024-03-26 01:54:52.274729414 +0000 UTC m=+0.285836698 container start 8ed421558ef9f3e7ae4c71a89a2dc723a4bd61983231292a8684760900ce059d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-26T01:54:52.345 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T01:54:52.345 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 7 2024-03-26T01:54:52.345 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: pidfile_write: ignore empty --pid-file 2024-03-26T01:54:52.345 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: load: jerasure load: lrc 2024-03-26T01:54:52.345 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: RocksDB version: 7.9.2 2024-03-26T01:54:52.345 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Git sha 0 2024-03-26T01:54:52.345 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T01:54:52.345 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: DB SUMMARY 2024-03-26T01:54:52.345 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: DB Session ID: EY85Y54TRFI4XX7OUTJN 2024-03-26T01:54:52.345 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: CURRENT file: CURRENT 2024-03-26T01:54:52.345 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T01:54:52.345 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-26T01:54:52.346 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-26T01:54:52.346 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84593 ; 2024-03-26T01:54:52.346 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.error_if_exists: 0 2024-03-26T01:54:52.346 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.create_if_missing: 0 2024-03-26T01:54:52.346 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.paranoid_checks: 1 2024-03-26T01:54:52.346 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T01:54:52.346 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T01:54:52.346 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T01:54:52.346 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.env: 0x55fb0ed80ae0 2024-03-26T01:54:52.346 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T01:54:52.346 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.info_log: 0x55fb10107920 2024-03-26T01:54:52.346 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T01:54:52.347 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.statistics: (nil) 2024-03-26T01:54:52.347 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.use_fsync: 0 2024-03-26T01:54:52.347 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_log_file_size: 0 2024-03-26T01:54:52.347 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T01:54:52.347 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T01:54:52.347 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T01:54:52.347 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T01:54:52.347 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.allow_fallocate: 1 2024-03-26T01:54:52.347 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T01:54:52.347 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T01:54:52.347 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.use_direct_reads: 0 2024-03-26T01:54:52.347 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T01:54:52.347 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T01:54:52.347 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.db_log_dir: 2024-03-26T01:54:52.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.wal_dir: 2024-03-26T01:54:52.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T01:54:52.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T01:54:52.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T01:54:52.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T01:54:52.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T01:54:52.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T01:54:52.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T01:54:52.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T01:54:52.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.write_buffer_manager: 0x55fb100da1e0 2024-03-26T01:54:52.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T01:54:52.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T01:54:52.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T01:54:52.349 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T01:54:52.349 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T01:54:52.349 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T01:54:52.350 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T01:54:52.350 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T01:54:52.350 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.unordered_write: 0 2024-03-26T01:54:52.350 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T01:54:52.350 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T01:54:52.350 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T01:54:52.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T01:54:52.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.row_cache: None 2024-03-26T01:54:52.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.wal_filter: None 2024-03-26T01:54:52.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T01:54:52.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T01:54:52.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.two_write_queues: 0 2024-03-26T01:54:52.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T01:54:52.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.wal_compression: 0 2024-03-26T01:54:52.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.atomic_flush: 0 2024-03-26T01:54:52.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T01:54:52.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T01:54:52.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T01:54:52.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.log_readahead_size: 0 2024-03-26T01:54:52.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T01:54:52.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T01:54:52.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T01:54:52.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T01:54:52.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T01:54:52.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T01:54:52.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T01:54:52.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_background_jobs: 2 2024-03-26T01:54:52.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_background_compactions: -1 2024-03-26T01:54:52.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_subcompactions: 1 2024-03-26T01:54:52.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T01:54:52.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T01:54:52.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T01:54:52.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T01:54:52.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T01:54:52.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T01:54:52.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T01:54:52.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T01:54:52.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_open_files: -1 2024-03-26T01:54:52.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T01:54:52.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T01:54:52.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T01:54:52.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T01:54:52.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_background_flushes: -1 2024-03-26T01:54:52.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Compression algorithms supported: 2024-03-26T01:54:52.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T01:54:52.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: kZSTD supported: 0 2024-03-26T01:54:52.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: kXpressCompression supported: 0 2024-03-26T01:54:52.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T01:54:52.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: kZlibCompression supported: 1 2024-03-26T01:54:52.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: kSnappyCompression supported: 1 2024-03-26T01:54:52.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: kLZ4Compression supported: 1 2024-03-26T01:54:52.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: kBZip2Compression supported: 0 2024-03-26T01:54:52.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T01:54:52.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T01:54:52.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-26T01:54:52.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T01:54:52.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T01:54:52.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.merge_operator: 2024-03-26T01:54:52.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compaction_filter: None 2024-03-26T01:54:52.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T01:54:52.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T01:54:52.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T01:54:52.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T01:54:52.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fb10107a20) 2024-03-26T01:54:52.355 INFO:journalctl@ceph.mon.a.smithi003.stdout: cache_index_and_filter_blocks: 1 2024-03-26T01:54:52.355 INFO:journalctl@ceph.mon.a.smithi003.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T01:54:52.355 INFO:journalctl@ceph.mon.a.smithi003.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T01:54:52.355 INFO:journalctl@ceph.mon.a.smithi003.stdout: pin_top_level_index_and_filter: 1 2024-03-26T01:54:52.355 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_type: 0 2024-03-26T01:54:52.355 INFO:journalctl@ceph.mon.a.smithi003.stdout: data_block_index_type: 0 2024-03-26T01:54:52.355 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_shortening: 1 2024-03-26T01:54:52.356 INFO:journalctl@ceph.mon.a.smithi003.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T01:54:52.356 INFO:journalctl@ceph.mon.a.smithi003.stdout: checksum: 4 2024-03-26T01:54:52.356 INFO:journalctl@ceph.mon.a.smithi003.stdout: no_block_cache: 0 2024-03-26T01:54:52.356 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache: 0x55fb1010b090 2024-03-26T01:54:52.356 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_name: BinnedLRUCache 2024-03-26T01:54:52.356 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_options: 2024-03-26T01:54:52.356 INFO:journalctl@ceph.mon.a.smithi003.stdout: capacity : 536870912 2024-03-26T01:54:52.356 INFO:journalctl@ceph.mon.a.smithi003.stdout: num_shard_bits : 4 2024-03-26T01:54:52.356 INFO:journalctl@ceph.mon.a.smithi003.stdout: strict_capacity_limit : 0 2024-03-26T01:54:52.356 INFO:journalctl@ceph.mon.a.smithi003.stdout: high_pri_pool_ratio: 0.000 2024-03-26T01:54:52.356 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_compressed: (nil) 2024-03-26T01:54:52.356 INFO:journalctl@ceph.mon.a.smithi003.stdout: persistent_cache: (nil) 2024-03-26T01:54:52.356 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_size: 4096 2024-03-26T01:54:52.356 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_size_deviation: 10 2024-03-26T01:54:52.357 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_restart_interval: 16 2024-03-26T01:54:52.357 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_block_restart_interval: 1 2024-03-26T01:54:52.357 INFO:journalctl@ceph.mon.a.smithi003.stdout: metadata_block_size: 4096 2024-03-26T01:54:52.357 INFO:journalctl@ceph.mon.a.smithi003.stdout: partition_filters: 0 2024-03-26T01:54:52.357 INFO:journalctl@ceph.mon.a.smithi003.stdout: use_delta_encoding: 1 2024-03-26T01:54:52.357 INFO:journalctl@ceph.mon.a.smithi003.stdout: filter_policy: bloomfilter 2024-03-26T01:54:52.357 INFO:journalctl@ceph.mon.a.smithi003.stdout: whole_key_filtering: 1 2024-03-26T01:54:52.357 INFO:journalctl@ceph.mon.a.smithi003.stdout: verify_compression: 0 2024-03-26T01:54:52.357 INFO:journalctl@ceph.mon.a.smithi003.stdout: read_amp_bytes_per_bit: 0 2024-03-26T01:54:52.357 INFO:journalctl@ceph.mon.a.smithi003.stdout: format_version: 5 2024-03-26T01:54:52.357 INFO:journalctl@ceph.mon.a.smithi003.stdout: enable_index_compression: 1 2024-03-26T01:54:52.357 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_align: 0 2024-03-26T01:54:52.357 INFO:journalctl@ceph.mon.a.smithi003.stdout: max_auto_readahead_size: 262144 2024-03-26T01:54:52.358 INFO:journalctl@ceph.mon.a.smithi003.stdout: prepopulate_block_cache: 0 2024-03-26T01:54:52.358 INFO:journalctl@ceph.mon.a.smithi003.stdout: initial_auto_readahead_size: 8192 2024-03-26T01:54:52.358 INFO:journalctl@ceph.mon.a.smithi003.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T01:54:52.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T01:54:52.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T01:54:52.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compression: NoCompression 2024-03-26T01:54:52.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T01:54:52.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T01:54:52.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T01:54:52.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.num_levels: 7 2024-03-26T01:54:52.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T01:54:52.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T01:54:52.359 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T01:54:52.359 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T01:54:52.359 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T01:54:52.359 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T01:54:52.359 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T01:54:52.359 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T01:54:52.359 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T01:54:52.359 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T01:54:52.359 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T01:54:52.359 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T01:54:52.359 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T01:54:52.359 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T01:54:52.359 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T01:54:52.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T01:54:52.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T01:54:52.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T01:54:52.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T01:54:52.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T01:54:52.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T01:54:52.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T01:54:52.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T01:54:52.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T01:54:52.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T01:54:52.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T01:54:52.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T01:54:52.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T01:54:52.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T01:54:52.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T01:54:52.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T01:54:52.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T01:54:52.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T01:54:52.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T01:54:52.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T01:54:52.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T01:54:52.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T01:54:52.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T01:54:52.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T01:54:52.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T01:54:52.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T01:54:52.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T01:54:52.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T01:54:52.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T01:54:52.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T01:54:52.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T01:54:52.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T01:54:52.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T01:54:52.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T01:54:52.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T01:54:52.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T01:54:52.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T01:54:52.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T01:54:52.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.table_properties_collectors: 2024-03-26T01:54:52.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.inplace_update_support: 0 2024-03-26T01:54:52.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T01:54:52.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T01:54:52.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T01:54:52.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T01:54:52.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.bloom_locality: 0 2024-03-26T01:54:52.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.max_successive_merges: 0 2024-03-26T01:54:52.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T01:54:52.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T01:54:52.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T01:54:52.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T01:54:52.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.ttl: 2592000 2024-03-26T01:54:52.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T01:54:52.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T01:54:52.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T01:54:52.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.enable_blob_files: false 2024-03-26T01:54:52.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.min_blob_size: 0 2024-03-26T01:54:52.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T01:54:52.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T01:54:52.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T01:54:52.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T01:54:52.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T01:54:52.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T01:54:52.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T01:54:52.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T01:54:52.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-26T01:54:52.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-26T01:54:52.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 71c9593f-7474-4eea-aed6-cc035385e321 2024-03-26T01:54:52.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418092303500, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-26T01:54:52.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-26T01:54:52.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418092304352, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81218, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79405, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73714, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711418092, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "71c9593f-7474-4eea-aed6-cc035385e321", "db_session_id": "EY85Y54TRFI4XX7OUTJN", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-26T01:54:52.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418092304438, "job": 1, "event": "recovery_finished"} 2024-03-26T01:54:52.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-26T01:54:52.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T01:54:52.366 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fb101fc000 2024-03-26T01:54:52.366 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: DB pointer 0x55fb101e6000 2024-03-26T01:54:52.366 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T01:54:52.366 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T01:54:52.366 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** DB Stats ** 2024-03-26T01:54:52.366 INFO:journalctl@ceph.mon.a.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T01:54:52.366 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T01:54:52.366 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T01:54:52.366 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T01:54:52.366 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T01:54:52.366 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T01:54:52.366 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T01:54:52.367 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-03-26T01:54:52.367 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2024-03-26T01:54:52.367 INFO:journalctl@ceph.mon.a.smithi003.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T01:54:52.367 INFO:journalctl@ceph.mon.a.smithi003.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T01:54:52.367 INFO:journalctl@ceph.mon.a.smithi003.stdout: L0 2/0 81.15 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 139.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T01:54:52.367 INFO:journalctl@ceph.mon.a.smithi003.stdout: Sum 2/0 81.15 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 139.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T01:54:52.367 INFO:journalctl@ceph.mon.a.smithi003.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 139.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T01:54:52.367 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-03-26T01:54:52.367 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2024-03-26T01:54:52.367 INFO:journalctl@ceph.mon.a.smithi003.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T01:54:52.367 INFO:journalctl@ceph.mon.a.smithi003.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T01:54:52.368 INFO:journalctl@ceph.mon.a.smithi003.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 139.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T01:54:52.368 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-03-26T01:54:52.368 INFO:journalctl@ceph.mon.a.smithi003.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T01:54:52.368 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-03-26T01:54:52.368 INFO:journalctl@ceph.mon.a.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T01:54:52.368 INFO:journalctl@ceph.mon.a.smithi003.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T01:54:52.368 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T01:54:52.368 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T01:54:52.368 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T01:54:52.368 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T01:54:52.368 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative compaction: 0.00 GB write, 26.89 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T01:54:52.368 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval compaction: 0.00 GB write, 26.89 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T01:54:52.368 INFO:journalctl@ceph.mon.a.smithi003.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T01:54:52.369 INFO:journalctl@ceph.mon.a.smithi003.stdout: Block cache BinnedLRUCache@0x55fb1010b090#7 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-26T01:54:52.369 INFO:journalctl@ceph.mon.a.smithi003.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-26T01:54:52.369 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-03-26T01:54:52.369 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T01:54:52.369 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: starting mon.a rank 0 at public addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] at bind addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T01:54:52.369 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: mon.a@-1(???) e1 preinit fsid b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T01:54:52.369 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: mon.a@-1(???).mds e1 new map 2024-03-26T01:54:52.369 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: mon.a@-1(???).mds e1 print_map 2024-03-26T01:54:52.369 INFO:journalctl@ceph.mon.a.smithi003.stdout: e1 2024-03-26T01:54:52.369 INFO:journalctl@ceph.mon.a.smithi003.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T01:54:52.369 INFO:journalctl@ceph.mon.a.smithi003.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T01:54:52.369 INFO:journalctl@ceph.mon.a.smithi003.stdout: legacy client fscid: -1 2024-03-26T01:54:52.369 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-03-26T01:54:52.369 INFO:journalctl@ceph.mon.a.smithi003.stdout: No filesystems configured 2024-03-26T01:54:52.370 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T01:54:52.370 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T01:54:52.370 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T01:54:52.370 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T01:54:52.370 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-26T01:54:52.370 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 bash[25772]: 8ed421558ef9f3e7ae4c71a89a2dc723a4bd61983231292a8684760900ce059d 2024-03-26T01:54:52.370 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 systemd[1]: Started Ceph mon.a for b95eae18-eb13-11ee-95d0-87774f69a715. 2024-03-26T01:54:52.370 INFO:teuthology.orchestra.run.smithi003.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-26T01:54:52.601 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T01:54:52.601 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: monmap e1: 1 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:54:52.601 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: fsmap 2024-03-26T01:54:52.601 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T01:54:52.601 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:52 smithi003 ceph-mon[25810]: mgrmap e1: no daemons active 2024-03-26T01:54:54.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:54:53 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/1467262526' entity='client.admin' 2024-03-26T01:54:55.689 INFO:teuthology.orchestra.run.smithi003.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-26T01:54:55.935 INFO:teuthology.orchestra.run.smithi003.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-26T01:54:55.935 INFO:teuthology.orchestra.run.smithi003.stdout:Creating mgr... 2024-03-26T01:54:55.935 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-26T01:54:55.936 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-26T01:54:57.015 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mgr.y 2024-03-26T01:54:57.015 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to reset failed state of unit ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mgr.y.service: Unit ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mgr.y.service not loaded. 2024-03-26T01:54:57.227 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-b95eae18-eb13-11ee-95d0-87774f69a715.target.wants/ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-b95eae18-eb13-11ee-95d0-87774f69a715@.service. 2024-03-26T01:54:57.390 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:54:57 smithi003 systemd[1]: Starting Ceph mgr.y for b95eae18-eb13-11ee-95d0-87774f69a715... 2024-03-26T01:54:57.891 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:54:57 smithi003 podman[26050]: 2024-03-26 01:54:57.361856314 +0000 UTC m=+0.017413819 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:54:58.640 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:54:58 smithi003 podman[26050]: 2024-03-26 01:54:58.24848355 +0000 UTC m=+0.904041046 container create e194b78ab17bbd40410c7f64566f054656f2c15a48a72200925a93a073b5e9ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-26T01:55:00.626 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:00 smithi003 podman[26050]: 2024-03-26 01:55:00.327474814 +0000 UTC m=+2.983032310 container init e194b78ab17bbd40410c7f64566f054656f2c15a48a72200925a93a073b5e9ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T01:55:00.626 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:00 smithi003 podman[26050]: 2024-03-26 01:55:00.335394689 +0000 UTC m=+2.990952186 container start e194b78ab17bbd40410c7f64566f054656f2c15a48a72200925a93a073b5e9ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-26T01:55:01.255 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:00 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:00.905+0000 7f1252cf0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T01:55:01.255 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:01 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:01.000+0000 7f1252cf0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T01:55:01.268 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T01:55:01.268 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2024-03-26T01:55:01.275 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T01:55:01.275 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-03-26T01:55:01.275 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2024-03-26T01:55:01.275 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T01:55:01.283 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T01:55:01.283 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2024-03-26T01:55:01.290 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T01:55:01.290 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-03-26T01:55:01.291 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2024-03-26T01:55:01.291 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-26T01:55:01.291 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr to start... 2024-03-26T01:55:01.291 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr... 2024-03-26T01:55:01.640 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:01 smithi003 bash[26050]: e194b78ab17bbd40410c7f64566f054656f2c15a48a72200925a93a073b5e9ad 2024-03-26T01:55:01.640 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:01 smithi003 systemd[1]: Started Ceph mgr.y for b95eae18-eb13-11ee-95d0-87774f69a715. 2024-03-26T01:55:02.010 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:01 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:01.671+0000 7f1252cf0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T01:55:02.010 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:01 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:01.766+0000 7f1252cf0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T01:55:02.010 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:01 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:01.909+0000 7f1252cf0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T01:55:02.010 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:02 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:02.006+0000 7f1252cf0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T01:55:02.280 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:02 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:02.092+0000 7f1252cf0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T01:55:02.561 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:02 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:02.360+0000 7f1252cf0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T01:55:02.561 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:02 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:02.472+0000 7f1252cf0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T01:55:02.562 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:02 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:02.561+0000 7f1252cf0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T01:55:02.847 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:02 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:02.657+0000 7f1252cf0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T01:55:02.847 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:02 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:02.846+0000 7f1252cf0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T01:55:03.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:03 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/779783335' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T01:55:03.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-26T01:55:03.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-03-26T01:55:03.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "b95eae18-eb13-11ee-95d0-87774f69a715", 2024-03-26T01:55:03.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T01:55:03.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T01:55:03.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T01:55:03.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T01:55:03.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:03.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T01:55:03.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T01:55:03.741 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2024-03-26T01:55:03.741 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-03-26T01:55:03.741 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T01:55:03.741 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T01:55:03.741 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-03-26T01:55:03.741 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-26T01:55:03.741 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T01:55:03.741 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:03.741 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T01:55:03.741 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T01:55:03.741 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:03.741 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T01:55:03.741 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:03.741 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T01:55:03.742 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T01:55:03.742 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T01:55:03.742 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T01:55:03.742 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T01:55:03.742 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T01:55:03.742 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:03.742 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T01:55:03.742 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T01:55:03.742 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T01:55:03.742 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T01:55:03.742 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T01:55:03.742 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T01:55:03.743 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T01:55:03.743 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T01:55:03.743 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T01:55:03.743 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:03.743 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T01:55:03.743 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:03.743 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T01:55:03.743 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T01:55:03.743 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:03.743 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T01:55:03.743 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T01:55:03.743 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T01:55:03.743 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T01:55:03.744 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T01:55:03.744 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T01:55:03.744 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T01:55:03.744 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-03-26T01:55:03.744 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T01:55:03.744 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:03.744 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T01:55:03.744 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:03.744 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T01:54:47.556204+0000", 2024-03-26T01:55:03.744 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T01:55:03.744 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:03.744 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T01:55:03.744 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-03-26T01:55:03.745 INFO:teuthology.orchestra.run.smithi003.stdout:mgr not available, waiting (1/15)... 2024-03-26T01:55:03.999 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:03 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:03.864+0000 7f1252cf0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T01:55:03.999 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:03 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:03.998+0000 7f1252cf0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T01:55:04.335 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:04 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:04.334+0000 7f1252cf0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T01:55:04.626 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:04 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:04.426+0000 7f1252cf0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T01:55:04.626 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:04 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:04.626+0000 7f1252cf0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:55:04.890 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:04 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:04.814+0000 7f1252cf0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T01:55:05.362 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:05 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:05.103+0000 7f1252cf0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:55:05.363 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:05 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:05.362+0000 7f1252cf0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T01:55:05.869 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:05 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:05.612+0000 7f1252cf0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T01:55:06.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:06 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/2514422260' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T01:55:06.640 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:06 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:06.527+0000 7f1252cf0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T01:55:06.946 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-26T01:55:06.946 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-03-26T01:55:06.946 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "b95eae18-eb13-11ee-95d0-87774f69a715", 2024-03-26T01:55:06.946 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T01:55:06.947 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T01:55:06.947 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T01:55:06.947 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T01:55:06.947 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:06.947 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T01:55:06.947 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T01:55:06.947 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2024-03-26T01:55:06.947 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-03-26T01:55:06.947 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T01:55:06.947 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T01:55:06.947 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-03-26T01:55:06.947 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-26T01:55:06.947 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T01:55:06.947 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:06.948 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T01:55:06.948 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T01:55:06.948 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:06.948 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T01:55:06.948 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:06.948 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T01:55:06.948 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T01:55:06.948 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T01:55:06.948 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T01:55:06.948 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T01:55:06.948 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T01:55:06.948 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:06.948 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T01:55:06.949 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T01:55:06.949 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T01:55:06.949 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T01:55:06.949 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T01:55:06.949 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T01:55:06.949 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T01:55:06.949 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T01:55:06.949 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T01:55:06.949 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:06.949 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T01:55:06.949 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:06.949 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T01:55:06.949 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T01:55:06.949 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:06.949 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T01:55:06.950 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T01:55:06.950 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T01:55:06.950 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T01:55:06.950 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T01:55:06.950 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T01:55:06.950 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T01:55:06.950 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-03-26T01:55:06.950 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T01:55:06.950 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:06.950 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T01:55:06.950 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:06.950 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T01:54:47.556204+0000", 2024-03-26T01:55:06.950 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T01:55:06.951 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:06.951 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T01:55:06.951 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-03-26T01:55:06.951 INFO:teuthology.orchestra.run.smithi003.stdout:mgr not available, waiting (2/15)... 2024-03-26T01:55:07.390 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:07 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:07.109+0000 7f1252cf0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T01:55:07.390 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:07 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:07.209+0000 7f1252cf0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T01:55:07.890 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:07 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:07.430+0000 7f1252cf0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T01:55:07.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:07 smithi003 ceph-mon[25810]: Activating manager daemon y 2024-03-26T01:55:07.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:07 smithi003 ceph-mon[25810]: mgrmap e2: y(active, starting, since 0.00306458s) 2024-03-26T01:55:07.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:07 smithi003 ceph-mon[25810]: from='mgr.14100 172.21.15.3:0/1324356735' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T01:55:07.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:07 smithi003 ceph-mon[25810]: from='mgr.14100 172.21.15.3:0/1324356735' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T01:55:07.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:07 smithi003 ceph-mon[25810]: from='mgr.14100 172.21.15.3:0/1324356735' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T01:55:07.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:07 smithi003 ceph-mon[25810]: from='mgr.14100 172.21.15.3:0/1324356735' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:55:07.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:07 smithi003 ceph-mon[25810]: from='mgr.14100 172.21.15.3:0/1324356735' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T01:55:07.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:07 smithi003 ceph-mon[25810]: Manager daemon y is now available 2024-03-26T01:55:07.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:07 smithi003 ceph-mon[25810]: from='mgr.14100 172.21.15.3:0/1324356735' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:55:07.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:07 smithi003 ceph-mon[25810]: from='mgr.14100 172.21.15.3:0/1324356735' entity='mgr.y' 2024-03-26T01:55:07.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:07 smithi003 ceph-mon[25810]: from='mgr.14100 172.21.15.3:0/1324356735' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T01:55:07.892 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:07 smithi003 ceph-mon[25810]: from='mgr.14100 172.21.15.3:0/1324356735' entity='mgr.y' 2024-03-26T01:55:07.892 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:07 smithi003 ceph-mon[25810]: from='mgr.14100 172.21.15.3:0/1324356735' entity='mgr.y' 2024-03-26T01:55:09.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:09 smithi003 ceph-mon[25810]: mgrmap e3: y(active, since 1.00753s) 2024-03-26T01:55:11.582 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:11 smithi003 ceph-mon[25810]: mgrmap e4: y(active, since 3s) 2024-03-26T01:55:11.583 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:11 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/3708816470' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T01:55:11.897 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-26T01:55:11.897 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-03-26T01:55:11.897 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "b95eae18-eb13-11ee-95d0-87774f69a715", 2024-03-26T01:55:11.897 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T01:55:11.898 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T01:55:11.898 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T01:55:11.898 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T01:55:11.898 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:11.898 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T01:55:11.898 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T01:55:11.898 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2024-03-26T01:55:11.898 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-03-26T01:55:11.898 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T01:55:11.898 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T01:55:11.898 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-03-26T01:55:11.898 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 18, 2024-03-26T01:55:11.898 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T01:55:11.898 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:11.899 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T01:55:11.899 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T01:55:11.899 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:11.899 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T01:55:11.899 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:11.899 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T01:55:11.899 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T01:55:11.899 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T01:55:11.899 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T01:55:11.899 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T01:55:11.899 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T01:55:11.903 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:11.903 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T01:55:11.903 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T01:55:11.903 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T01:55:11.903 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T01:55:11.903 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T01:55:11.903 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T01:55:11.903 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T01:55:11.904 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T01:55:11.904 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T01:55:11.904 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:11.904 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T01:55:11.904 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:11.904 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T01:55:11.904 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T01:55:11.904 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:11.904 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T01:55:11.904 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T01:55:11.904 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T01:55:11.904 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T01:55:11.904 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T01:55:11.904 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T01:55:11.905 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T01:55:11.905 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-03-26T01:55:11.905 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T01:55:11.905 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:11.905 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T01:55:11.905 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:11.905 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T01:54:47.556204+0000", 2024-03-26T01:55:11.905 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T01:55:11.905 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:11.905 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T01:55:11.905 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-03-26T01:55:11.905 INFO:teuthology.orchestra.run.smithi003.stdout:mgr is available 2024-03-26T01:55:12.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:12 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/1968397256' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T01:55:12.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:12 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/1968397256' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T01:55:13.026 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-26T01:55:13.026 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T01:55:13.026 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout fsid = b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T01:55:13.026 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T01:55:13.026 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T01:55:13.026 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T01:55:13.026 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T01:55:13.026 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-26T01:55:13.026 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T01:55:13.026 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T01:55:13.026 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-26T01:55:13.027 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T01:55:13.027 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T01:55:13.027 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T01:55:13.027 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling cephadm module... 2024-03-26T01:55:14.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:13 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/4160253646' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-26T01:55:14.102 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:13 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: ignoring --setuser ceph since I am not root 2024-03-26T01:55:14.103 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:13 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: ignoring --setgroup ceph since I am not root 2024-03-26T01:55:14.621 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:14 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:14.366+0000 7f80407bd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T01:55:15.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:14 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:14.468+0000 7f80407bd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T01:55:15.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:14 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/4160253646' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-26T01:55:15.025 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:14 smithi003 ceph-mon[25810]: mgrmap e5: y(active, since 6s) 2024-03-26T01:55:15.390 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:15 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:15.138+0000 7f80407bd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T01:55:15.390 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:15 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:15.238+0000 7f80407bd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T01:55:15.390 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:15 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:15.384+0000 7f80407bd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T01:55:15.791 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:15 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:15.483+0000 7f80407bd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T01:55:15.791 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:15 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:15.570+0000 7f80407bd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T01:55:16.050 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:15 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/231681998' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T01:55:16.050 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:15 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:15.841+0000 7f80407bd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T01:55:16.050 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:15 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:15.955+0000 7f80407bd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T01:55:16.050 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:16 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:16.049+0000 7f80407bd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T01:55:16.087 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-03-26T01:55:16.087 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-26T01:55:16.088 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T01:55:16.088 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T01:55:16.088 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T01:55:16.088 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-03-26T01:55:16.088 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for the mgr to restart... 2024-03-26T01:55:16.088 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr epoch 5... 2024-03-26T01:55:16.334 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:16 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:16.151+0000 7f80407bd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T01:55:16.334 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:16 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:16.332+0000 7f80407bd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T01:55:17.640 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:17 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:17.336+0000 7f80407bd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T01:55:17.641 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:17 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:17.472+0000 7f80407bd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T01:55:18.133 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:17 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:17.833+0000 7f80407bd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T01:55:18.133 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:17 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:17.927+0000 7f80407bd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T01:55:18.133 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:18 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:18.132+0000 7f80407bd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:55:18.390 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:18 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:18.325+0000 7f80407bd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T01:55:18.885 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:18 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:18.623+0000 7f80407bd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:55:18.885 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:18 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:18.884+0000 7f80407bd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T01:55:19.140 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:19 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:19.130+0000 7f80407bd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T01:55:20.390 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:19 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:19.968+0000 7f80407bd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T01:55:20.873 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:20 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:20.550+0000 7f80407bd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T01:55:20.873 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:20 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:20.650+0000 7f80407bd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T01:55:20.873 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:20 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:20.872+0000 7f80407bd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T01:55:21.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:20 smithi003 ceph-mon[25810]: Active manager daemon y restarted 2024-03-26T01:55:21.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:20 smithi003 ceph-mon[25810]: Activating manager daemon y 2024-03-26T01:55:21.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:20 smithi003 ceph-mon[25810]: osdmap e2: 0 total, 0 up, 0 in 2024-03-26T01:55:21.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:20 smithi003 ceph-mon[25810]: mgrmap e6: y(active, starting, since 0.00392038s) 2024-03-26T01:55:21.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:20 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:55:21.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:20 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T01:55:21.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:20 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T01:55:21.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:20 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T01:55:21.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:20 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T01:55:21.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:20 smithi003 ceph-mon[25810]: Manager daemon y is now available 2024-03-26T01:55:21.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:20 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' 2024-03-26T01:55:21.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:20 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' 2024-03-26T01:55:21.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:20 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:55:21.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:20 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:55:22.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:21 smithi003 ceph-mon[25810]: Found migration_current of "None". Setting to last migration. 2024-03-26T01:55:22.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:21 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:55:22.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:21 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:55:22.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:21 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T01:55:22.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:21 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' 2024-03-26T01:55:22.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:21 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' 2024-03-26T01:55:22.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:21 smithi003 ceph-mon[25810]: mgrmap e7: y(active, since 1.00546s) 2024-03-26T01:55:22.527 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-03-26T01:55:22.527 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-26T01:55:22.527 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T01:55:22.527 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-03-26T01:55:22.527 INFO:teuthology.orchestra.run.smithi003.stdout:mgr epoch 5 is available 2024-03-26T01:55:22.527 INFO:teuthology.orchestra.run.smithi003.stdout:Setting orchestrator backend to cephadm... 2024-03-26T01:55:23.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:22 smithi003 ceph-mon[25810]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T01:55:23.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:22 smithi003 ceph-mon[25810]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T01:55:23.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:22 smithi003 ceph-mon[25810]: [26/Mar/2024:01:55:21] ENGINE Bus STARTING 2024-03-26T01:55:23.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:22 smithi003 ceph-mon[25810]: [26/Mar/2024:01:55:22] ENGINE Serving on https://172.21.15.3:7150 2024-03-26T01:55:23.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:22 smithi003 ceph-mon[25810]: [26/Mar/2024:01:55:22] ENGINE Serving on http://172.21.15.3:8765 2024-03-26T01:55:23.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:22 smithi003 ceph-mon[25810]: [26/Mar/2024:01:55:22] ENGINE Bus STARTED 2024-03-26T01:55:23.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:22 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:55:24.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:23 smithi003 ceph-mon[25810]: mgrmap e8: y(active, since 2s) 2024-03-26T01:55:24.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:23 smithi003 ceph-mon[25810]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:24.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:23 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' 2024-03-26T01:55:24.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:23 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:55:24.946 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-26T01:55:24.946 INFO:teuthology.orchestra.run.smithi003.stdout:Generating ssh key... 2024-03-26T01:55:25.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:24 smithi003 ceph-mon[25810]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:26.124 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:25 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: Generating public/private rsa key pair. 2024-03-26T01:55:26.124 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:25 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: Your identification has been saved in /tmp/tmpjm0gn_zq/key. 2024-03-26T01:55:26.124 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:25 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: Your public key has been saved in /tmp/tmpjm0gn_zq/key.pub. 2024-03-26T01:55:26.124 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:25 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: The key fingerprint is: 2024-03-26T01:55:26.124 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:25 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: SHA256:zXkj2tG5qj6rngA086Tgb2TrY9bv/UWhduvZEmHnacs ceph-b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T01:55:26.124 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:25 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: The key's randomart image is: 2024-03-26T01:55:26.124 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:25 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: +---[RSA 3072]----+ 2024-03-26T01:55:26.125 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:25 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: | | 2024-03-26T01:55:26.125 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:25 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: | | 2024-03-26T01:55:26.125 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:25 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: |. + . . | 2024-03-26T01:55:26.125 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:25 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: |.o * o o.+.. | 2024-03-26T01:55:26.125 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:25 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: | .oo. S *o*o+ .| 2024-03-26T01:55:26.125 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:25 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: | +.. o.+o+.+ | 2024-03-26T01:55:26.125 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:25 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: | +o . . .o+ .| 2024-03-26T01:55:26.125 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:25 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: | o+ o .o .o.oE | 2024-03-26T01:55:26.125 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:25 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: | o...*=+=o. o.. | 2024-03-26T01:55:26.125 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:25 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: +----[SHA256]-----+ 2024-03-26T01:55:27.081 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:26 smithi003 ceph-mon[25810]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:27.081 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:26 smithi003 ceph-mon[25810]: Generating ssh key... 2024-03-26T01:55:27.081 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:26 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' 2024-03-26T01:55:27.081 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:26 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' 2024-03-26T01:55:27.565 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCo6BPfmf4H5FxMBBecFBaP95tG+NgH/mzE9HjJtIw8d39tOql4rrrLDSd+hNbe+s3SM3tXc+gZeOjSm3dxofUwGFT2DKbOhDPG6NSamOg1S+Nz5H88OQ0RvGU5+wrPd0qsKymd9GdwdB2mvTIRgjtrPxHESFRUgBoodd5PsW6iY55BaZU56FAcco1maw5kFnC/B39RakROMfXJnUA1TDQ17EhRD76PpcWF5bjuqg8ZWDpmZj2/oYlpQWOryiwUt7ewo3LM/I/rBzxHAjiGjiNvFtOsWJWDvPmslKMCHBSZv3X7FwadknqboUrl9SH7+b3QMfqet8el5yKA3Dsn54Zv9G5hUZ+lqZO1FIsBAb7yVOZe6GxiDYjwsPrVxIBvhGj+xcinsC4VXK3paFUOV6x3Oyp7Rgn6cRbH1vfB+wUxZillQL/bBpU2xObwrXuXvztLph5rfabNYiMts81zl4WBK0ZI0obYjyBOX8Ygg4GItQaHiIr3EyuN42WK1pOEKv0= ceph-b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T01:55:27.566 INFO:teuthology.orchestra.run.smithi003.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-26T01:55:27.566 INFO:teuthology.orchestra.run.smithi003.stdout:Adding key to root@localhost authorized_keys... 2024-03-26T01:55:27.566 INFO:teuthology.orchestra.run.smithi003.stdout:Adding host smithi003... 2024-03-26T01:55:28.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:27 smithi003 ceph-mon[25810]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:29.114 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:28 smithi003 ceph-mon[25810]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi003", "addr": "172.21.15.3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:30.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:29 smithi003 ceph-mon[25810]: Deploying cephadm binary to smithi003 2024-03-26T01:55:33.214 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Added host 'smithi003' with addr '172.21.15.3' 2024-03-26T01:55:33.214 INFO:teuthology.orchestra.run.smithi003.stdout:Deploying unmanaged mon service... 2024-03-26T01:55:33.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:33 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' 2024-03-26T01:55:33.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:33 smithi003 ceph-mon[25810]: Added host smithi003 2024-03-26T01:55:33.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:33 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:55:34.547 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-26T01:55:34.547 INFO:teuthology.orchestra.run.smithi003.stdout:Deploying unmanaged mgr service... 2024-03-26T01:55:35.082 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:34 smithi003 ceph-mon[25810]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:35.082 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:34 smithi003 ceph-mon[25810]: Saving service mon spec with placement count:5 2024-03-26T01:55:35.083 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:34 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' 2024-03-26T01:55:36.215 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-26T01:55:36.783 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:36 smithi003 ceph-mon[25810]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:36.783 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:36 smithi003 ceph-mon[25810]: Saving service mgr spec with placement count:2 2024-03-26T01:55:36.783 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:36 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' 2024-03-26T01:55:38.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:37 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' 2024-03-26T01:55:38.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:37 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/3943308677' entity='client.admin' 2024-03-26T01:55:38.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:37 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' 2024-03-26T01:55:39.205 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling the dashboard module... 2024-03-26T01:55:39.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:39 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/1502763816' entity='client.admin' 2024-03-26T01:55:41.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:40 smithi003 ceph-mon[25810]: from='mgr.14118 172.21.15.3:0/2511873958' entity='mgr.y' 2024-03-26T01:55:41.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:40 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/606920114' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-26T01:55:41.236 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:40 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: ignoring --setuser ceph since I am not root 2024-03-26T01:55:41.236 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:40 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: ignoring --setgroup ceph since I am not root 2024-03-26T01:55:41.770 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:41 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:41.500+0000 7f6b8f5b4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T01:55:41.770 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:41 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:41.595+0000 7f6b8f5b4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T01:55:42.039 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:41 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/606920114' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-26T01:55:42.039 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:41 smithi003 ceph-mon[25810]: mgrmap e9: y(active, since 20s) 2024-03-26T01:55:42.640 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:42 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:42.310+0000 7f6b8f5b4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T01:55:42.640 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:42 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:42.405+0000 7f6b8f5b4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T01:55:42.640 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:42 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:42.554+0000 7f6b8f5b4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T01:55:42.918 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:42 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:42.652+0000 7f6b8f5b4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T01:55:42.918 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:42 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:42.738+0000 7f6b8f5b4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T01:55:43.208 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:42 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/2883192328' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T01:55:43.209 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:43 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:43.007+0000 7f6b8f5b4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T01:55:43.209 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:43 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:43.120+0000 7f6b8f5b4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T01:55:43.209 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:43 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:43.209+0000 7f6b8f5b4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T01:55:43.320 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-03-26T01:55:43.321 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-26T01:55:43.321 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T01:55:43.321 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T01:55:43.321 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T01:55:43.321 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-03-26T01:55:43.321 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for the mgr to restart... 2024-03-26T01:55:43.321 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr epoch 9... 2024-03-26T01:55:43.490 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:43 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:43.306+0000 7f6b8f5b4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T01:55:43.890 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:43 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:43.490+0000 7f6b8f5b4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T01:55:44.890 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:44 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:44.483+0000 7f6b8f5b4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T01:55:44.890 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:44 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:44.618+0000 7f6b8f5b4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T01:55:45.259 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:44 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:44.966+0000 7f6b8f5b4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T01:55:45.259 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:45 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:45.057+0000 7f6b8f5b4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T01:55:45.259 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:45 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:45.259+0000 7f6b8f5b4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:55:45.640 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:45 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:45.450+0000 7f6b8f5b4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T01:55:46.003 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:45 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:45.742+0000 7f6b8f5b4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:55:46.003 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:46 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:46.003+0000 7f6b8f5b4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T01:55:46.390 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:46 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:46.246+0000 7f6b8f5b4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T01:55:47.390 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:47 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:47.072+0000 7f6b8f5b4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T01:55:47.980 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:47 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:47.651+0000 7f6b8f5b4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T01:55:47.980 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:47 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:47.757+0000 7f6b8f5b4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T01:55:47.980 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:55:47 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:55:47.980+0000 7f6b8f5b4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T01:55:48.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:48 smithi003 ceph-mon[25810]: Active manager daemon y restarted 2024-03-26T01:55:48.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:48 smithi003 ceph-mon[25810]: Activating manager daemon y 2024-03-26T01:55:48.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:48 smithi003 ceph-mon[25810]: osdmap e3: 0 total, 0 up, 0 in 2024-03-26T01:55:48.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:48 smithi003 ceph-mon[25810]: mgrmap e10: y(active, starting, since 0.00316373s) 2024-03-26T01:55:48.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:48 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:55:48.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:48 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T01:55:48.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:48 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T01:55:48.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:48 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T01:55:48.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:48 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T01:55:48.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:48 smithi003 ceph-mon[25810]: Manager daemon y is now available 2024-03-26T01:55:48.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:48 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:55:48.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:48 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:55:49.327 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:49 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:55:49.327 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:49 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T01:55:49.327 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:49 smithi003 ceph-mon[25810]: [26/Mar/2024:01:55:48] ENGINE Bus STARTING 2024-03-26T01:55:49.327 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:49 smithi003 ceph-mon[25810]: [26/Mar/2024:01:55:48] ENGINE Serving on https://172.21.15.3:7150 2024-03-26T01:55:49.327 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:49 smithi003 ceph-mon[25810]: [26/Mar/2024:01:55:48] ENGINE Serving on http://172.21.15.3:8765 2024-03-26T01:55:49.327 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:49 smithi003 ceph-mon[25810]: [26/Mar/2024:01:55:48] ENGINE Bus STARTED 2024-03-26T01:55:49.327 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:49 smithi003 ceph-mon[25810]: mgrmap e11: y(active, since 1.00576s) 2024-03-26T01:55:50.171 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-03-26T01:55:50.171 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-26T01:55:50.171 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T01:55:50.171 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-03-26T01:55:50.171 INFO:teuthology.orchestra.run.smithi003.stdout:mgr epoch 9 is available 2024-03-26T01:55:50.171 INFO:teuthology.orchestra.run.smithi003.stdout:Generating a dashboard self-signed certificate... 2024-03-26T01:55:51.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:50 smithi003 ceph-mon[25810]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T01:55:51.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:50 smithi003 ceph-mon[25810]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T01:55:51.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:50 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:55:51.676 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-26T01:55:51.676 INFO:teuthology.orchestra.run.smithi003.stdout:Creating initial admin user... 2024-03-26T01:55:52.101 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:51 smithi003 ceph-mon[25810]: mgrmap e12: y(active, since 2s) 2024-03-26T01:55:52.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:51 smithi003 ceph-mon[25810]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:52.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:55:52.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:55:52.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:55:53.444 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$hfkAkIUE./UKjH/ZwKGMtub3KTpR8rqa/BDjKVLH4SE2unRnQKxyC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711418152, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-26T01:55:53.444 INFO:teuthology.orchestra.run.smithi003.stdout:Fetching dashboard port number... 2024-03-26T01:55:53.779 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:53 smithi003 ceph-mon[25810]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:53.779 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:53 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:55:54.873 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:54 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/2433170222' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-26T01:55:55.135 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 8443 2024-03-26T01:55:55.145 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T01:55:55.145 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2024-03-26T01:55:55.152 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T01:55:55.152 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-03-26T01:55:55.152 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2024-03-26T01:55:55.152 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-26T01:55:55.154 INFO:teuthology.orchestra.run.smithi003.stdout:Ceph Dashboard is now available at: 2024-03-26T01:55:55.154 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:55:55.154 INFO:teuthology.orchestra.run.smithi003.stdout: URL: https://smithi003.front.sepia.ceph.com:8443/ 2024-03-26T01:55:55.155 INFO:teuthology.orchestra.run.smithi003.stdout: User: admin 2024-03-26T01:55:55.155 INFO:teuthology.orchestra.run.smithi003.stdout: Password: pf9e793vaj 2024-03-26T01:55:55.155 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:55:55.155 INFO:teuthology.orchestra.run.smithi003.stdout:Saving cluster configuration to /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config directory 2024-03-26T01:55:55.155 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling autotune for osd_memory_target 2024-03-26T01:55:55.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:55 smithi003 ceph-mon[25810]: mgrmap e13: y(active, since 6s) 2024-03-26T01:55:57.822 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-26T01:55:57.822 INFO:teuthology.orchestra.run.smithi003.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-26T01:55:57.822 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:55:57.822 INFO:teuthology.orchestra.run.smithi003.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-26T01:55:57.822 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:55:57.822 INFO:teuthology.orchestra.run.smithi003.stdout:Or, if you are only running a single cluster on this host: 2024-03-26T01:55:57.823 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:55:57.823 INFO:teuthology.orchestra.run.smithi003.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-26T01:55:57.823 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:55:57.823 INFO:teuthology.orchestra.run.smithi003.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-26T01:55:57.823 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:55:57.823 INFO:teuthology.orchestra.run.smithi003.stdout: ceph telemetry on 2024-03-26T01:55:57.823 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:55:57.823 INFO:teuthology.orchestra.run.smithi003.stdout:For more information see: 2024-03-26T01:55:57.823 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:55:57.823 INFO:teuthology.orchestra.run.smithi003.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-26T01:55:57.823 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:55:57.823 INFO:teuthology.orchestra.run.smithi003.stdout:Bootstrap complete. 2024-03-26T01:55:57.866 INFO:tasks.cephadm:Fetching config... 2024-03-26T01:55:57.867 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-26T01:55:57.867 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-26T01:55:57.882 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-26T01:55:57.882 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-26T01:55:57.882 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-26T01:55:57.938 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-26T01:55:57.938 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-26T01:55:57.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-26T01:55:58.006 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-26T01:55:58.006 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-26T01:55:58.006 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-26T01:55:58.061 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-26T01:55:58.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCo6BPfmf4H5FxMBBecFBaP95tG+NgH/mzE9HjJtIw8d39tOql4rrrLDSd+hNbe+s3SM3tXc+gZeOjSm3dxofUwGFT2DKbOhDPG6NSamOg1S+Nz5H88OQ0RvGU5+wrPd0qsKymd9GdwdB2mvTIRgjtrPxHESFRUgBoodd5PsW6iY55BaZU56FAcco1maw5kFnC/B39RakROMfXJnUA1TDQ17EhRD76PpcWF5bjuqg8ZWDpmZj2/oYlpQWOryiwUt7ewo3LM/I/rBzxHAjiGjiNvFtOsWJWDvPmslKMCHBSZv3X7FwadknqboUrl9SH7+b3QMfqet8el5yKA3Dsn54Zv9G5hUZ+lqZO1FIsBAb7yVOZe6GxiDYjwsPrVxIBvhGj+xcinsC4VXK3paFUOV6x3Oyp7Rgn6cRbH1vfB+wUxZillQL/bBpU2xObwrXuXvztLph5rfabNYiMts81zl4WBK0ZI0obYjyBOX8Ygg4GItQaHiIr3EyuN42WK1pOEKv0= ceph-b95eae18-eb13-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T01:55:58.140 INFO:teuthology.orchestra.run.smithi003.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCo6BPfmf4H5FxMBBecFBaP95tG+NgH/mzE9HjJtIw8d39tOql4rrrLDSd+hNbe+s3SM3tXc+gZeOjSm3dxofUwGFT2DKbOhDPG6NSamOg1S+Nz5H88OQ0RvGU5+wrPd0qsKymd9GdwdB2mvTIRgjtrPxHESFRUgBoodd5PsW6iY55BaZU56FAcco1maw5kFnC/B39RakROMfXJnUA1TDQ17EhRD76PpcWF5bjuqg8ZWDpmZj2/oYlpQWOryiwUt7ewo3LM/I/rBzxHAjiGjiNvFtOsWJWDvPmslKMCHBSZv3X7FwadknqboUrl9SH7+b3QMfqet8el5yKA3Dsn54Zv9G5hUZ+lqZO1FIsBAb7yVOZe6GxiDYjwsPrVxIBvhGj+xcinsC4VXK3paFUOV6x3Oyp7Rgn6cRbH1vfB+wUxZillQL/bBpU2xObwrXuXvztLph5rfabNYiMts81zl4WBK0ZI0obYjyBOX8Ygg4GItQaHiIr3EyuN42WK1pOEKv0= ceph-b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T01:55:58.154 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCo6BPfmf4H5FxMBBecFBaP95tG+NgH/mzE9HjJtIw8d39tOql4rrrLDSd+hNbe+s3SM3tXc+gZeOjSm3dxofUwGFT2DKbOhDPG6NSamOg1S+Nz5H88OQ0RvGU5+wrPd0qsKymd9GdwdB2mvTIRgjtrPxHESFRUgBoodd5PsW6iY55BaZU56FAcco1maw5kFnC/B39RakROMfXJnUA1TDQ17EhRD76PpcWF5bjuqg8ZWDpmZj2/oYlpQWOryiwUt7ewo3LM/I/rBzxHAjiGjiNvFtOsWJWDvPmslKMCHBSZv3X7FwadknqboUrl9SH7+b3QMfqet8el5yKA3Dsn54Zv9G5hUZ+lqZO1FIsBAb7yVOZe6GxiDYjwsPrVxIBvhGj+xcinsC4VXK3paFUOV6x3Oyp7Rgn6cRbH1vfB+wUxZillQL/bBpU2xObwrXuXvztLph5rfabNYiMts81zl4WBK0ZI0obYjyBOX8Ygg4GItQaHiIr3EyuN42WK1pOEKv0= ceph-b95eae18-eb13-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T01:55:58.199 INFO:teuthology.orchestra.run.smithi055.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCo6BPfmf4H5FxMBBecFBaP95tG+NgH/mzE9HjJtIw8d39tOql4rrrLDSd+hNbe+s3SM3tXc+gZeOjSm3dxofUwGFT2DKbOhDPG6NSamOg1S+Nz5H88OQ0RvGU5+wrPd0qsKymd9GdwdB2mvTIRgjtrPxHESFRUgBoodd5PsW6iY55BaZU56FAcco1maw5kFnC/B39RakROMfXJnUA1TDQ17EhRD76PpcWF5bjuqg8ZWDpmZj2/oYlpQWOryiwUt7ewo3LM/I/rBzxHAjiGjiNvFtOsWJWDvPmslKMCHBSZv3X7FwadknqboUrl9SH7+b3QMfqet8el5yKA3Dsn54Zv9G5hUZ+lqZO1FIsBAb7yVOZe6GxiDYjwsPrVxIBvhGj+xcinsC4VXK3paFUOV6x3Oyp7Rgn6cRbH1vfB+wUxZillQL/bBpU2xObwrXuXvztLph5rfabNYiMts81zl4WBK0ZI0obYjyBOX8Ygg4GItQaHiIr3EyuN42WK1pOEKv0= ceph-b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T01:55:58.214 DEBUG:teuthology.orchestra.run.smithi066:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCo6BPfmf4H5FxMBBecFBaP95tG+NgH/mzE9HjJtIw8d39tOql4rrrLDSd+hNbe+s3SM3tXc+gZeOjSm3dxofUwGFT2DKbOhDPG6NSamOg1S+Nz5H88OQ0RvGU5+wrPd0qsKymd9GdwdB2mvTIRgjtrPxHESFRUgBoodd5PsW6iY55BaZU56FAcco1maw5kFnC/B39RakROMfXJnUA1TDQ17EhRD76PpcWF5bjuqg8ZWDpmZj2/oYlpQWOryiwUt7ewo3LM/I/rBzxHAjiGjiNvFtOsWJWDvPmslKMCHBSZv3X7FwadknqboUrl9SH7+b3QMfqet8el5yKA3Dsn54Zv9G5hUZ+lqZO1FIsBAb7yVOZe6GxiDYjwsPrVxIBvhGj+xcinsC4VXK3paFUOV6x3Oyp7Rgn6cRbH1vfB+wUxZillQL/bBpU2xObwrXuXvztLph5rfabNYiMts81zl4WBK0ZI0obYjyBOX8Ygg4GItQaHiIr3EyuN42WK1pOEKv0= ceph-b95eae18-eb13-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T01:55:58.267 INFO:teuthology.orchestra.run.smithi066.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCo6BPfmf4H5FxMBBecFBaP95tG+NgH/mzE9HjJtIw8d39tOql4rrrLDSd+hNbe+s3SM3tXc+gZeOjSm3dxofUwGFT2DKbOhDPG6NSamOg1S+Nz5H88OQ0RvGU5+wrPd0qsKymd9GdwdB2mvTIRgjtrPxHESFRUgBoodd5PsW6iY55BaZU56FAcco1maw5kFnC/B39RakROMfXJnUA1TDQ17EhRD76PpcWF5bjuqg8ZWDpmZj2/oYlpQWOryiwUt7ewo3LM/I/rBzxHAjiGjiNvFtOsWJWDvPmslKMCHBSZv3X7FwadknqboUrl9SH7+b3QMfqet8el5yKA3Dsn54Zv9G5hUZ+lqZO1FIsBAb7yVOZe6GxiDYjwsPrVxIBvhGj+xcinsC4VXK3paFUOV6x3Oyp7Rgn6cRbH1vfB+wUxZillQL/bBpU2xObwrXuXvztLph5rfabNYiMts81zl4WBK0ZI0obYjyBOX8Ygg4GItQaHiIr3EyuN42WK1pOEKv0= ceph-b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T01:55:58.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-26T01:55:58.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:55:58 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/1213136207' entity='client.admin' 2024-03-26T01:55:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:56:01.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:00 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:01.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:00 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:01.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:00 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi003", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:56:01.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:00 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:01.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:00 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:01.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:00 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:01.598 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-26T01:56:01.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-26T01:56:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:56:02.210 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:02 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/2448253130' entity='client.admin' 2024-03-26T01:56:02.211 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:02 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:02.211 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:02 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:04.339 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi055 2024-03-26T01:56:04.340 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T01:56:04.340 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/etc/ceph/ceph.conf 2024-03-26T01:56:04.357 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T01:56:04.358 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:56:04.415 INFO:tasks.cephadm:Adding host smithi055 to orchestrator... 2024-03-26T01:56:04.416 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph orch host add smithi055 2024-03-26T01:56:04.653 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:56:04.874 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:04 smithi003 ceph-mon[25810]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:04.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:04 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:04.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:04 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:04.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:04 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:04.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:04 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:56:04.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:04 smithi003 ceph-mon[25810]: Updating smithi003:/etc/ceph/ceph.conf 2024-03-26T01:56:04.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:04 smithi003 ceph-mon[25810]: Updating smithi003:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:04.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:04 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:04.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:04 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:04.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:04 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:05.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:05 smithi003 ceph-mon[25810]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:56:05.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:05 smithi003 ceph-mon[25810]: Updating smithi003:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:56:07.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:07 smithi003 ceph-mon[25810]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi055", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:08.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:08 smithi003 ceph-mon[25810]: Deploying cephadm binary to smithi055 2024-03-26T01:56:09.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:09 smithi003 ceph-mon[25810]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:10.885 INFO:teuthology.orchestra.run.smithi003.stdout:Added host 'smithi055' with addr '172.21.15.55' 2024-03-26T01:56:11.528 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T01:56:11.771 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:56:12.134 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:11 smithi003 ceph-mon[25810]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:12.134 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:11 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:12.134 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:11 smithi003 ceph-mon[25810]: Added host smithi055 2024-03-26T01:56:12.134 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:11 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:12.134 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:11 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:13.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:12 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:13.527 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:56:13.527 INFO:teuthology.orchestra.run.smithi003.stdout:[{"addr": "172.21.15.3", "hostname": "smithi003", "labels": [], "status": ""}, {"addr": "172.21.15.55", "hostname": "smithi055", "labels": [], "status": ""}] 2024-03-26T01:56:14.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:13 smithi003 ceph-mon[25810]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:14.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:13 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:14.172 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi066 2024-03-26T01:56:14.172 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T01:56:14.172 DEBUG:teuthology.orchestra.run.smithi066:> dd of=/etc/ceph/ceph.conf 2024-03-26T01:56:14.193 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T01:56:14.193 DEBUG:teuthology.orchestra.run.smithi066:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:56:14.252 INFO:tasks.cephadm:Adding host smithi066 to orchestrator... 2024-03-26T01:56:14.252 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph orch host add smithi066 2024-03-26T01:56:14.489 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:56:15.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:14 smithi003 ceph-mon[25810]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:56:16.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:15 smithi003 ceph-mon[25810]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:18.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:17 smithi003 ceph-mon[25810]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:18.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:17 smithi003 ceph-mon[25810]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi066", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:18.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:17 smithi003 ceph-mon[25810]: Deploying cephadm binary to smithi066 2024-03-26T01:56:18.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:17 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:18.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:17 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:18.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:17 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:18.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:17 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:18.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:17 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi055", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:56:18.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:17 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:18.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:17 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:56:18.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:17 smithi003 ceph-mon[25810]: Updating smithi055:/etc/ceph/ceph.conf 2024-03-26T01:56:19.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:18 smithi003 ceph-mon[25810]: Updating smithi055:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:19.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:18 smithi003 ceph-mon[25810]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:56:20.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:19 smithi003 ceph-mon[25810]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:20.735 INFO:teuthology.orchestra.run.smithi003.stdout:Added host 'smithi066' with addr '172.21.15.66' 2024-03-26T01:56:21.064 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:20 smithi003 ceph-mon[25810]: Updating smithi055:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:56:21.064 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:20 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:21.064 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:20 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:21.064 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:20 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:21.064 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:20 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:21.064 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:20 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:21.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T01:56:21.577 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:56:22.034 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:21 smithi003 ceph-mon[25810]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:22.034 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:21 smithi003 ceph-mon[25810]: Added host smithi066 2024-03-26T01:56:22.034 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:21 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:22.034 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:21 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:23.361 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:56:23.361 INFO:teuthology.orchestra.run.smithi003.stdout:[{"addr": "172.21.15.3", "hostname": "smithi003", "labels": [], "status": ""}, {"addr": "172.21.15.55", "hostname": "smithi055", "labels": [], "status": ""}, {"addr": "172.21.15.66", "hostname": "smithi066", "labels": [], "status": ""}] 2024-03-26T01:56:24.062 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-26T01:56:24.063 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-26T01:56:24.298 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:56:24.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:23 smithi003 ceph-mon[25810]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:24.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:23 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:25.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:24 smithi003 ceph-mon[25810]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:56:26.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:25 smithi003 ceph-mon[25810]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:26.948 INFO:teuthology.orchestra.run.smithi003.stderr:adjusted tunables profile to default 2024-03-26T01:56:27.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:26 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/2525554678' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T01:56:27.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:26 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:27.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:26 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:27.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:26 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:27.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:26 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:27.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:26 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi066", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:56:27.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:26 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:27.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:26 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:56:27.652 INFO:tasks.cephadm:Adding mon.a on smithi003 2024-03-26T01:56:27.652 INFO:tasks.cephadm:Adding mon.b on smithi055 2024-03-26T01:56:27.652 INFO:tasks.cephadm:Adding mon.c on smithi066 2024-03-26T01:56:27.652 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi003:172.21.15.3=a;smithi055:172.21.15.55=b;smithi066:172.21.15.66=c' 2024-03-26T01:56:27.890 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:28.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:27 smithi003 ceph-mon[25810]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:28.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:27 smithi003 ceph-mon[25810]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-26T01:56:28.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:27 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/2525554678' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T01:56:28.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:27 smithi003 ceph-mon[25810]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:56:28.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:27 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:28.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:27 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:28.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:27 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:29.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:28 smithi003 ceph-mon[25810]: Updating smithi066:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:29.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:28 smithi003 ceph-mon[25810]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:56:29.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:28 smithi003 ceph-mon[25810]: Updating smithi066:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:56:29.606 INFO:teuthology.orchestra.run.smithi066.stdout:Scheduled mon update... 2024-03-26T01:56:30.299 DEBUG:teuthology.orchestra.run.smithi055:mon.b> sudo journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mon.b.service 2024-03-26T01:56:30.301 DEBUG:teuthology.orchestra.run.smithi066:mon.c> sudo journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mon.c.service 2024-03-26T01:56:30.304 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T01:56:30.304 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T01:56:30.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:29 smithi003 ceph-mon[25810]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:30.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:29 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:30.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:29 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:30.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:29 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:30.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:29 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:56:30.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:29 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:30.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:29 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:56:30.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:29 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:31.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:30 smithi003 ceph-mon[25810]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi003:172.21.15.3=a;smithi055:172.21.15.55=b;smithi066:172.21.15.66=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:31.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:30 smithi003 ceph-mon[25810]: Saving service mon spec with placement smithi003:172.21.15.3=a;smithi055:172.21.15.55=b;smithi066:172.21.15.66=c;count:3 2024-03-26T01:56:31.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:30 smithi003 ceph-mon[25810]: Deploying daemon mon.c on smithi066 2024-03-26T01:56:32.091 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:32.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:31 smithi003 ceph-mon[25810]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:33.841 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:33 smithi066 systemd[1]: Starting Ceph mon.c for b95eae18-eb13-11ee-95d0-87774f69a715... 2024-03-26T01:56:34.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 podman[30173]: 2024-03-26 01:56:34.090752804 +0000 UTC m=+0.153077324 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T01:56:34.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 podman[30173]: 2024-03-26 01:56:34.218398915 +0000 UTC m=+0.280723435 container create 8b36477ce6de8e88839efb4eeede4f153992258ae0d41f554a7589c2b0f92b5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-c, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T01:56:34.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:33 smithi003 ceph-mon[25810]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:34.686 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 podman[30173]: 2024-03-26 01:56:34.629826109 +0000 UTC m=+0.692150629 container init 8b36477ce6de8e88839efb4eeede4f153992258ae0d41f554a7589c2b0f92b5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-26T01:56:34.687 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 podman[30173]: 2024-03-26 01:56:34.636072414 +0000 UTC m=+0.698396938 container start 8b36477ce6de8e88839efb4eeede4f153992258ae0d41f554a7589c2b0f92b5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-5e70124) 2024-03-26T01:56:34.687 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T01:56:34.687 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 7 2024-03-26T01:56:34.687 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: pidfile_write: ignore empty --pid-file 2024-03-26T01:56:34.687 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: load: jerasure load: lrc 2024-03-26T01:56:34.687 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: RocksDB version: 7.9.2 2024-03-26T01:56:34.687 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Git sha 0 2024-03-26T01:56:34.687 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T01:56:34.687 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: DB SUMMARY 2024-03-26T01:56:34.687 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: DB Session ID: TZ1FNNDOWYK1Y0UTW8N8 2024-03-26T01:56:34.687 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: CURRENT file: CURRENT 2024-03-26T01:56:34.687 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T01:56:34.688 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-26T01:56:34.688 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-26T01:56:34.688 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-26T01:56:34.688 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.error_if_exists: 0 2024-03-26T01:56:34.688 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.create_if_missing: 0 2024-03-26T01:56:34.688 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.paranoid_checks: 1 2024-03-26T01:56:34.688 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T01:56:34.688 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T01:56:34.688 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T01:56:34.688 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.env: 0x563a43dcfae0 2024-03-26T01:56:34.688 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T01:56:34.688 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.info_log: 0x563a449ad920 2024-03-26T01:56:34.688 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T01:56:34.689 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.statistics: (nil) 2024-03-26T01:56:34.689 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.use_fsync: 0 2024-03-26T01:56:34.689 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_log_file_size: 0 2024-03-26T01:56:34.690 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T01:56:34.690 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T01:56:34.690 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T01:56:34.690 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T01:56:34.690 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.allow_fallocate: 1 2024-03-26T01:56:34.690 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T01:56:34.690 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T01:56:34.690 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.use_direct_reads: 0 2024-03-26T01:56:34.690 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T01:56:34.690 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T01:56:34.690 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.db_log_dir: 2024-03-26T01:56:34.690 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.wal_dir: 2024-03-26T01:56:34.690 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T01:56:34.691 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T01:56:34.691 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T01:56:34.691 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T01:56:34.691 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T01:56:34.691 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T01:56:34.691 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T01:56:34.692 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T01:56:34.692 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.write_buffer_manager: 0x563a449801e0 2024-03-26T01:56:34.692 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T01:56:34.692 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T01:56:34.692 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T01:56:34.692 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T01:56:34.692 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T01:56:34.692 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T01:56:34.692 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T01:56:34.692 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T01:56:34.692 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.unordered_write: 0 2024-03-26T01:56:34.692 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T01:56:34.692 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T01:56:34.693 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T01:56:34.693 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T01:56:34.693 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.row_cache: None 2024-03-26T01:56:34.693 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.wal_filter: None 2024-03-26T01:56:34.693 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T01:56:34.693 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T01:56:34.693 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.two_write_queues: 0 2024-03-26T01:56:34.693 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T01:56:34.693 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.wal_compression: 0 2024-03-26T01:56:34.693 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.atomic_flush: 0 2024-03-26T01:56:34.693 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T01:56:34.693 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T01:56:34.693 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T01:56:34.694 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.log_readahead_size: 0 2024-03-26T01:56:34.694 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T01:56:34.694 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T01:56:34.694 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T01:56:34.694 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T01:56:34.694 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T01:56:34.694 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T01:56:34.694 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T01:56:34.694 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_background_jobs: 2 2024-03-26T01:56:34.694 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_background_compactions: -1 2024-03-26T01:56:34.694 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_subcompactions: 1 2024-03-26T01:56:34.694 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T01:56:34.694 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T01:56:34.694 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T01:56:34.695 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T01:56:34.695 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T01:56:34.695 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T01:56:34.695 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T01:56:34.695 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T01:56:34.695 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_open_files: -1 2024-03-26T01:56:34.695 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T01:56:34.695 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T01:56:34.695 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T01:56:34.695 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T01:56:34.695 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_background_flushes: -1 2024-03-26T01:56:34.695 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Compression algorithms supported: 2024-03-26T01:56:34.695 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T01:56:34.695 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: kZSTD supported: 0 2024-03-26T01:56:34.696 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: kXpressCompression supported: 0 2024-03-26T01:56:34.696 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T01:56:34.696 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: kZlibCompression supported: 1 2024-03-26T01:56:34.696 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: kSnappyCompression supported: 1 2024-03-26T01:56:34.696 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: kLZ4Compression supported: 1 2024-03-26T01:56:34.696 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: kBZip2Compression supported: 0 2024-03-26T01:56:34.696 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T01:56:34.696 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T01:56:34.696 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-26T01:56:34.696 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T01:56:34.696 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T01:56:34.696 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.merge_operator: 2024-03-26T01:56:34.696 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compaction_filter: None 2024-03-26T01:56:34.697 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T01:56:34.697 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T01:56:34.697 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T01:56:34.697 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T01:56:34.697 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563a449ada20) 2024-03-26T01:56:34.697 INFO:journalctl@ceph.mon.c.smithi066.stdout: cache_index_and_filter_blocks: 1 2024-03-26T01:56:34.697 INFO:journalctl@ceph.mon.c.smithi066.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T01:56:34.697 INFO:journalctl@ceph.mon.c.smithi066.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T01:56:34.697 INFO:journalctl@ceph.mon.c.smithi066.stdout: pin_top_level_index_and_filter: 1 2024-03-26T01:56:34.697 INFO:journalctl@ceph.mon.c.smithi066.stdout: index_type: 0 2024-03-26T01:56:34.697 INFO:journalctl@ceph.mon.c.smithi066.stdout: data_block_index_type: 0 2024-03-26T01:56:34.697 INFO:journalctl@ceph.mon.c.smithi066.stdout: index_shortening: 1 2024-03-26T01:56:34.697 INFO:journalctl@ceph.mon.c.smithi066.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T01:56:34.698 INFO:journalctl@ceph.mon.c.smithi066.stdout: checksum: 4 2024-03-26T01:56:34.698 INFO:journalctl@ceph.mon.c.smithi066.stdout: no_block_cache: 0 2024-03-26T01:56:34.698 INFO:journalctl@ceph.mon.c.smithi066.stdout: block_cache: 0x563a449b1090 2024-03-26T01:56:34.698 INFO:journalctl@ceph.mon.c.smithi066.stdout: block_cache_name: BinnedLRUCache 2024-03-26T01:56:34.698 INFO:journalctl@ceph.mon.c.smithi066.stdout: block_cache_options: 2024-03-26T01:56:34.698 INFO:journalctl@ceph.mon.c.smithi066.stdout: capacity : 536870912 2024-03-26T01:56:34.698 INFO:journalctl@ceph.mon.c.smithi066.stdout: num_shard_bits : 4 2024-03-26T01:56:34.698 INFO:journalctl@ceph.mon.c.smithi066.stdout: strict_capacity_limit : 0 2024-03-26T01:56:34.698 INFO:journalctl@ceph.mon.c.smithi066.stdout: high_pri_pool_ratio: 0.000 2024-03-26T01:56:34.698 INFO:journalctl@ceph.mon.c.smithi066.stdout: block_cache_compressed: (nil) 2024-03-26T01:56:34.698 INFO:journalctl@ceph.mon.c.smithi066.stdout: persistent_cache: (nil) 2024-03-26T01:56:34.698 INFO:journalctl@ceph.mon.c.smithi066.stdout: block_size: 4096 2024-03-26T01:56:34.698 INFO:journalctl@ceph.mon.c.smithi066.stdout: block_size_deviation: 10 2024-03-26T01:56:34.698 INFO:journalctl@ceph.mon.c.smithi066.stdout: block_restart_interval: 16 2024-03-26T01:56:34.699 INFO:journalctl@ceph.mon.c.smithi066.stdout: index_block_restart_interval: 1 2024-03-26T01:56:34.699 INFO:journalctl@ceph.mon.c.smithi066.stdout: metadata_block_size: 4096 2024-03-26T01:56:34.699 INFO:journalctl@ceph.mon.c.smithi066.stdout: partition_filters: 0 2024-03-26T01:56:34.699 INFO:journalctl@ceph.mon.c.smithi066.stdout: use_delta_encoding: 1 2024-03-26T01:56:34.699 INFO:journalctl@ceph.mon.c.smithi066.stdout: filter_policy: bloomfilter 2024-03-26T01:56:34.699 INFO:journalctl@ceph.mon.c.smithi066.stdout: whole_key_filtering: 1 2024-03-26T01:56:34.699 INFO:journalctl@ceph.mon.c.smithi066.stdout: verify_compression: 0 2024-03-26T01:56:34.699 INFO:journalctl@ceph.mon.c.smithi066.stdout: read_amp_bytes_per_bit: 0 2024-03-26T01:56:34.699 INFO:journalctl@ceph.mon.c.smithi066.stdout: format_version: 5 2024-03-26T01:56:34.699 INFO:journalctl@ceph.mon.c.smithi066.stdout: enable_index_compression: 1 2024-03-26T01:56:34.699 INFO:journalctl@ceph.mon.c.smithi066.stdout: block_align: 0 2024-03-26T01:56:34.699 INFO:journalctl@ceph.mon.c.smithi066.stdout: max_auto_readahead_size: 262144 2024-03-26T01:56:34.699 INFO:journalctl@ceph.mon.c.smithi066.stdout: prepopulate_block_cache: 0 2024-03-26T01:56:34.699 INFO:journalctl@ceph.mon.c.smithi066.stdout: initial_auto_readahead_size: 8192 2024-03-26T01:56:34.700 INFO:journalctl@ceph.mon.c.smithi066.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T01:56:34.700 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T01:56:34.700 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T01:56:34.700 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compression: NoCompression 2024-03-26T01:56:34.700 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T01:56:34.700 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T01:56:34.700 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T01:56:34.700 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.num_levels: 7 2024-03-26T01:56:34.700 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T01:56:34.700 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T01:56:34.700 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T01:56:34.700 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T01:56:34.700 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T01:56:34.700 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T01:56:34.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T01:56:34.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T01:56:34.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T01:56:34.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T01:56:34.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T01:56:34.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T01:56:34.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T01:56:34.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T01:56:34.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T01:56:34.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T01:56:34.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T01:56:34.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T01:56:34.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T01:56:34.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T01:56:34.702 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T01:56:34.702 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T01:56:34.702 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T01:56:34.702 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T01:56:34.702 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T01:56:34.702 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T01:56:34.702 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T01:56:34.702 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T01:56:34.702 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T01:56:34.702 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T01:56:34.702 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T01:56:34.703 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T01:56:34.704 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T01:56:34.704 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T01:56:34.704 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T01:56:34.704 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T01:56:34.704 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T01:56:34.704 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T01:56:34.704 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T01:56:34.704 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T01:56:34.704 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T01:56:34.704 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T01:56:34.704 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T01:56:34.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T01:56:34.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T01:56:34.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T01:56:34.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T01:56:34.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T01:56:34.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T01:56:34.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T01:56:34.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T01:56:34.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T01:56:34.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T01:56:34.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.table_properties_collectors: 2024-03-26T01:56:34.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.inplace_update_support: 0 2024-03-26T01:56:34.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T01:56:34.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T01:56:34.706 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T01:56:34.706 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T01:56:34.706 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.bloom_locality: 0 2024-03-26T01:56:34.706 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.max_successive_merges: 0 2024-03-26T01:56:34.706 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T01:56:34.706 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T01:56:34.706 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T01:56:34.706 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T01:56:34.706 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.ttl: 2592000 2024-03-26T01:56:34.706 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T01:56:34.706 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T01:56:34.706 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T01:56:34.706 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.enable_blob_files: false 2024-03-26T01:56:34.706 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.min_blob_size: 0 2024-03-26T01:56:34.707 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T01:56:34.707 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T01:56:34.707 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T01:56:34.707 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T01:56:34.707 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T01:56:34.707 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T01:56:34.707 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T01:56:34.707 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T01:56:34.707 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T01:56:34.707 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-26T01:56:34.707 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: cf1806c1-cdaa-4d6c-9dc6-4f0fe2d4aa60 2024-03-26T01:56:34.707 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418194666605, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-26T01:56:34.707 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-26T01:56:34.708 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418194667135, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711418194, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "cf1806c1-cdaa-4d6c-9dc6-4f0fe2d4aa60", "db_session_id": "TZ1FNNDOWYK1Y0UTW8N8", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-26T01:56:34.708 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418194667239, "job": 1, "event": "recovery_finished"} 2024-03-26T01:56:34.708 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-26T01:56:34.708 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T01:56:34.708 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563a44aa2000 2024-03-26T01:56:34.708 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: DB pointer 0x563a44a8c000 2024-03-26T01:56:34.708 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T01:56:34.708 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T01:56:34.708 INFO:journalctl@ceph.mon.c.smithi066.stdout: ** DB Stats ** 2024-03-26T01:56:34.708 INFO:journalctl@ceph.mon.c.smithi066.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T01:56:34.708 INFO:journalctl@ceph.mon.c.smithi066.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T01:56:34.708 INFO:journalctl@ceph.mon.c.smithi066.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T01:56:34.708 INFO:journalctl@ceph.mon.c.smithi066.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T01:56:34.709 INFO:journalctl@ceph.mon.c.smithi066.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T01:56:34.709 INFO:journalctl@ceph.mon.c.smithi066.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T01:56:34.709 INFO:journalctl@ceph.mon.c.smithi066.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T01:56:34.709 INFO:journalctl@ceph.mon.c.smithi066.stdout: 2024-03-26T01:56:34.709 INFO:journalctl@ceph.mon.c.smithi066.stdout: ** Compaction Stats [default] ** 2024-03-26T01:56:34.709 INFO:journalctl@ceph.mon.c.smithi066.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T01:56:34.709 INFO:journalctl@ceph.mon.c.smithi066.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T01:56:34.709 INFO:journalctl@ceph.mon.c.smithi066.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T01:56:34.709 INFO:journalctl@ceph.mon.c.smithi066.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T01:56:34.709 INFO:journalctl@ceph.mon.c.smithi066.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T01:56:34.709 INFO:journalctl@ceph.mon.c.smithi066.stdout: 2024-03-26T01:56:34.709 INFO:journalctl@ceph.mon.c.smithi066.stdout: ** Compaction Stats [default] ** 2024-03-26T01:56:34.709 INFO:journalctl@ceph.mon.c.smithi066.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T01:56:34.710 INFO:journalctl@ceph.mon.c.smithi066.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T01:56:34.710 INFO:journalctl@ceph.mon.c.smithi066.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T01:56:34.710 INFO:journalctl@ceph.mon.c.smithi066.stdout: 2024-03-26T01:56:34.710 INFO:journalctl@ceph.mon.c.smithi066.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T01:56:34.710 INFO:journalctl@ceph.mon.c.smithi066.stdout: 2024-03-26T01:56:34.710 INFO:journalctl@ceph.mon.c.smithi066.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T01:56:34.710 INFO:journalctl@ceph.mon.c.smithi066.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T01:56:34.710 INFO:journalctl@ceph.mon.c.smithi066.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T01:56:34.710 INFO:journalctl@ceph.mon.c.smithi066.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T01:56:34.710 INFO:journalctl@ceph.mon.c.smithi066.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T01:56:34.710 INFO:journalctl@ceph.mon.c.smithi066.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T01:56:34.710 INFO:journalctl@ceph.mon.c.smithi066.stdout: Cumulative compaction: 0.00 GB write, 0.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T01:56:34.710 INFO:journalctl@ceph.mon.c.smithi066.stdout: Interval compaction: 0.00 GB write, 0.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T01:56:34.711 INFO:journalctl@ceph.mon.c.smithi066.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T01:56:34.711 INFO:journalctl@ceph.mon.c.smithi066.stdout: Block cache BinnedLRUCache@0x563a449b1090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-26T01:56:34.711 INFO:journalctl@ceph.mon.c.smithi066.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-26T01:56:34.711 INFO:journalctl@ceph.mon.c.smithi066.stdout: 2024-03-26T01:56:34.711 INFO:journalctl@ceph.mon.c.smithi066.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T01:56:34.711 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-26T01:56:34.711 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: using public_addr v2:172.21.15.66:0/0 -> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] 2024-03-26T01:56:34.711 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: starting mon.c rank -1 at public addrs [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] at bind addrs [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T01:56:34.711 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: mon.c@-1(???) e0 preinit fsid b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T01:56:34.711 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: mon.c@-1(synchronizing).mds e1 new map 2024-03-26T01:56:34.711 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-26T01:56:34.711 INFO:journalctl@ceph.mon.c.smithi066.stdout: e1 2024-03-26T01:56:34.711 INFO:journalctl@ceph.mon.c.smithi066.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T01:56:34.711 INFO:journalctl@ceph.mon.c.smithi066.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T01:56:34.712 INFO:journalctl@ceph.mon.c.smithi066.stdout: legacy client fscid: -1 2024-03-26T01:56:34.712 INFO:journalctl@ceph.mon.c.smithi066.stdout: 2024-03-26T01:56:34.712 INFO:journalctl@ceph.mon.c.smithi066.stdout: No filesystems configured 2024-03-26T01:56:34.712 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-26T01:56:34.712 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-26T01:56:34.712 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-26T01:56:34.712 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-26T01:56:34.712 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-26T01:56:34.712 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-26T01:56:34.712 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T01:56:34.713 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T01:56:34.713 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T01:56:34.713 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T01:56:34.713 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:56:34.713 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:34.713 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:34.713 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi066", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:34.713 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: Deploying cephadm binary to smithi066 2024-03-26T01:56:34.713 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:34.713 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:34.713 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:34.713 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:34.713 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi055", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:56:34.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:34.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:56:34.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: Updating smithi055:/etc/ceph/ceph.conf 2024-03-26T01:56:34.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: Updating smithi055:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:34.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:56:34.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:34.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: Updating smithi055:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:56:34.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:34.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:34.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:34.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:34.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:34.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:34.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: Added host smithi066 2024-03-26T01:56:34.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:34.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:34.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:34.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:34.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:56:34.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:34.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/2525554678' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T01:56:34.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:34.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:34.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:34.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:34.716 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi066", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:56:34.716 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:34.716 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:56:34.716 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:34.716 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-26T01:56:34.716 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/2525554678' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T01:56:34.716 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:56:34.716 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:34.716 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:34.716 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:34.716 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: Updating smithi066:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:34.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:56:34.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: Updating smithi066:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:56:34.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:34.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:34.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:34.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:34.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:56:34.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:34.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:56:34.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:34.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi003:172.21.15.3=a;smithi055:172.21.15.55=b;smithi066:172.21.15.66=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:34.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: Saving service mon spec with placement smithi003:172.21.15.3=a;smithi055:172.21.15.55=b;smithi066:172.21.15.66=c;count:3 2024-03-26T01:56:34.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: Deploying daemon mon.c on smithi066 2024-03-26T01:56:34.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:34.718 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:34.718 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 ceph-mon[30259]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-26T01:56:34.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 bash[30173]: 8b36477ce6de8e88839efb4eeede4f153992258ae0d41f554a7589c2b0f92b5d 2024-03-26T01:56:34.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:34 smithi066 systemd[1]: Started Ceph mon.c for b95eae18-eb13-11ee-95d0-87774f69a715. 2024-03-26T01:56:39.741 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T01:56:39.741 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":2,"fsid":"b95eae18-eb13-11ee-95d0-87774f69a715","modified":"2024-03-26T01:56:34.693046Z","created":"2024-03-26T01:54:44.578800Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:3300","nonce":0},{"type":"v1","addr":"172.21.15.66:6789","nonce":0}]},"addr":"172.21.15.66:6789/0","public_addr":"172.21.15.66:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T01:56:39.744 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 2 2024-03-26T01:56:40.024 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:40.025 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:56:40.025 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: mon.a calling monitor election 2024-03-26T01:56:40.025 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:40.025 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: from='client.? 172.21.15.66:0/1440825932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:56:40.025 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:40.025 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:40.025 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:40.025 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: mon.c calling monitor election 2024-03-26T01:56:40.025 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:40.025 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:40.025 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:40.025 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:40.025 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T01:56:40.025 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: monmap e2: 2 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],c=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:56:40.026 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: fsmap 2024-03-26T01:56:40.026 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:56:40.026 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: mgrmap e13: y(active, since 51s) 2024-03-26T01:56:40.026 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: overall HEALTH_OK 2024-03-26T01:56:40.026 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:40.026 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:40.026 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:40.026 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:56:40.026 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:39 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:40.052 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:40.052 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:56:40.052 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: mon.a calling monitor election 2024-03-26T01:56:40.052 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:40.052 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: from='client.? 172.21.15.66:0/1440825932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:56:40.052 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:40.052 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:40.052 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:40.053 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: mon.c calling monitor election 2024-03-26T01:56:40.053 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:40.053 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:40.053 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:40.053 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:40.053 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T01:56:40.053 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: monmap e2: 2 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],c=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:56:40.053 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: fsmap 2024-03-26T01:56:40.053 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:56:40.053 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: mgrmap e13: y(active, since 51s) 2024-03-26T01:56:40.053 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: overall HEALTH_OK 2024-03-26T01:56:40.053 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:40.053 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:40.054 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:40.054 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:56:40.054 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:39 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:41.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:40 smithi003 ceph-mon[25810]: Deploying daemon mon.b on smithi055 2024-03-26T01:56:41.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:40 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:41.221 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T01:56:41.221 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T01:56:41.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:40 smithi066 ceph-mon[30259]: Deploying daemon mon.b on smithi055 2024-03-26T01:56:41.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:40 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:41.480 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T01:56:41.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:41 smithi066 ceph-mon[30259]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:42.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:41 smithi003 ceph-mon[25810]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:43.343 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T01:56:43.344 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":2,"fsid":"b95eae18-eb13-11ee-95d0-87774f69a715","modified":"2024-03-26T01:56:34.693046Z","created":"2024-03-26T01:54:44.578800Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:3300","nonce":0},{"type":"v1","addr":"172.21.15.66:6789","nonce":0}]},"addr":"172.21.15.66:6789/0","public_addr":"172.21.15.66:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T01:56:43.345 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 2 2024-03-26T01:56:43.448 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 podman[30580]: 2024-03-26 01:56:43.264959545 +0000 UTC m=+0.017696328 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T01:56:43.771 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 podman[30580]: 2024-03-26 01:56:43.518406238 +0000 UTC m=+0.271143009 container init 6085b4c176b300748781c9cf6b2fced119c6f9d35fe10a0b18e83cb6b0a61e19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.build-date=20240319) 2024-03-26T01:56:43.771 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 podman[30580]: 2024-03-26 01:56:43.524075932 +0000 UTC m=+0.276812703 container start 6085b4c176b300748781c9cf6b2fced119c6f9d35fe10a0b18e83cb6b0a61e19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T01:56:43.772 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T01:56:43.772 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 7 2024-03-26T01:56:43.772 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: pidfile_write: ignore empty --pid-file 2024-03-26T01:56:43.772 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: load: jerasure load: lrc 2024-03-26T01:56:43.772 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: RocksDB version: 7.9.2 2024-03-26T01:56:43.772 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Git sha 0 2024-03-26T01:56:43.773 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T01:56:43.773 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: DB SUMMARY 2024-03-26T01:56:43.773 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: DB Session ID: HGPP249KPP2P2JBH56FX 2024-03-26T01:56:43.773 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: CURRENT file: CURRENT 2024-03-26T01:56:43.773 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T01:56:43.773 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-26T01:56:43.774 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-26T01:56:43.774 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-26T01:56:43.774 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.error_if_exists: 0 2024-03-26T01:56:43.774 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.create_if_missing: 0 2024-03-26T01:56:43.774 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.paranoid_checks: 1 2024-03-26T01:56:43.774 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T01:56:43.775 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T01:56:43.775 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T01:56:43.775 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.env: 0x55600e110ae0 2024-03-26T01:56:43.775 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T01:56:43.775 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.info_log: 0x55600fd19920 2024-03-26T01:56:43.776 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T01:56:43.776 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.statistics: (nil) 2024-03-26T01:56:43.776 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.use_fsync: 0 2024-03-26T01:56:43.776 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_log_file_size: 0 2024-03-26T01:56:43.776 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T01:56:43.776 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T01:56:43.777 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T01:56:43.777 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T01:56:43.777 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.allow_fallocate: 1 2024-03-26T01:56:43.777 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T01:56:43.777 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T01:56:43.777 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.use_direct_reads: 0 2024-03-26T01:56:43.778 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T01:56:43.778 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T01:56:43.778 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.db_log_dir: 2024-03-26T01:56:43.778 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.wal_dir: 2024-03-26T01:56:43.778 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T01:56:43.778 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T01:56:43.779 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T01:56:43.779 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T01:56:43.779 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T01:56:43.779 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T01:56:43.779 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T01:56:43.779 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T01:56:43.780 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.write_buffer_manager: 0x55600fcec1e0 2024-03-26T01:56:43.780 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T01:56:43.780 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T01:56:43.780 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T01:56:43.780 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T01:56:43.780 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T01:56:43.781 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T01:56:43.781 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T01:56:43.781 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T01:56:43.781 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.unordered_write: 0 2024-03-26T01:56:43.781 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T01:56:43.781 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T01:56:43.782 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T01:56:43.782 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T01:56:43.782 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.row_cache: None 2024-03-26T01:56:43.782 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.wal_filter: None 2024-03-26T01:56:43.782 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T01:56:43.782 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T01:56:43.783 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.two_write_queues: 0 2024-03-26T01:56:43.783 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T01:56:43.783 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.wal_compression: 0 2024-03-26T01:56:43.783 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.atomic_flush: 0 2024-03-26T01:56:43.783 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T01:56:43.783 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T01:56:43.784 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T01:56:43.784 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.log_readahead_size: 0 2024-03-26T01:56:43.784 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T01:56:43.784 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T01:56:43.784 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T01:56:43.784 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T01:56:43.785 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T01:56:43.785 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T01:56:43.785 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T01:56:43.785 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_background_jobs: 2 2024-03-26T01:56:43.785 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_background_compactions: -1 2024-03-26T01:56:43.785 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_subcompactions: 1 2024-03-26T01:56:43.786 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T01:56:43.786 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T01:56:43.786 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T01:56:43.786 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T01:56:43.786 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T01:56:43.786 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T01:56:43.787 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T01:56:43.787 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T01:56:43.787 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_open_files: -1 2024-03-26T01:56:43.787 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T01:56:43.787 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T01:56:43.787 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T01:56:43.788 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T01:56:43.788 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_background_flushes: -1 2024-03-26T01:56:43.788 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Compression algorithms supported: 2024-03-26T01:56:43.788 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T01:56:43.788 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: kZSTD supported: 0 2024-03-26T01:56:43.788 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: kXpressCompression supported: 0 2024-03-26T01:56:43.789 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T01:56:43.789 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: kZlibCompression supported: 1 2024-03-26T01:56:43.789 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: kSnappyCompression supported: 1 2024-03-26T01:56:43.789 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: kLZ4Compression supported: 1 2024-03-26T01:56:43.789 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: kBZip2Compression supported: 0 2024-03-26T01:56:43.789 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T01:56:43.790 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T01:56:43.790 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-26T01:56:43.790 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T01:56:43.790 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T01:56:43.790 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.merge_operator: 2024-03-26T01:56:43.790 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compaction_filter: None 2024-03-26T01:56:43.791 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T01:56:43.791 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T01:56:43.791 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T01:56:43.791 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T01:56:43.791 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55600fd19a20) 2024-03-26T01:56:43.791 INFO:journalctl@ceph.mon.b.smithi055.stdout: cache_index_and_filter_blocks: 1 2024-03-26T01:56:43.792 INFO:journalctl@ceph.mon.b.smithi055.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T01:56:43.792 INFO:journalctl@ceph.mon.b.smithi055.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T01:56:43.792 INFO:journalctl@ceph.mon.b.smithi055.stdout: pin_top_level_index_and_filter: 1 2024-03-26T01:56:43.792 INFO:journalctl@ceph.mon.b.smithi055.stdout: index_type: 0 2024-03-26T01:56:43.792 INFO:journalctl@ceph.mon.b.smithi055.stdout: data_block_index_type: 0 2024-03-26T01:56:43.792 INFO:journalctl@ceph.mon.b.smithi055.stdout: index_shortening: 1 2024-03-26T01:56:43.793 INFO:journalctl@ceph.mon.b.smithi055.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T01:56:43.793 INFO:journalctl@ceph.mon.b.smithi055.stdout: checksum: 4 2024-03-26T01:56:43.793 INFO:journalctl@ceph.mon.b.smithi055.stdout: no_block_cache: 0 2024-03-26T01:56:43.793 INFO:journalctl@ceph.mon.b.smithi055.stdout: block_cache: 0x55600fd1d090 2024-03-26T01:56:43.793 INFO:journalctl@ceph.mon.b.smithi055.stdout: block_cache_name: BinnedLRUCache 2024-03-26T01:56:43.793 INFO:journalctl@ceph.mon.b.smithi055.stdout: block_cache_options: 2024-03-26T01:56:43.793 INFO:journalctl@ceph.mon.b.smithi055.stdout: capacity : 536870912 2024-03-26T01:56:43.794 INFO:journalctl@ceph.mon.b.smithi055.stdout: num_shard_bits : 4 2024-03-26T01:56:43.794 INFO:journalctl@ceph.mon.b.smithi055.stdout: strict_capacity_limit : 0 2024-03-26T01:56:43.794 INFO:journalctl@ceph.mon.b.smithi055.stdout: high_pri_pool_ratio: 0.000 2024-03-26T01:56:43.794 INFO:journalctl@ceph.mon.b.smithi055.stdout: block_cache_compressed: (nil) 2024-03-26T01:56:43.794 INFO:journalctl@ceph.mon.b.smithi055.stdout: persistent_cache: (nil) 2024-03-26T01:56:43.795 INFO:journalctl@ceph.mon.b.smithi055.stdout: block_size: 4096 2024-03-26T01:56:43.795 INFO:journalctl@ceph.mon.b.smithi055.stdout: block_size_deviation: 10 2024-03-26T01:56:43.795 INFO:journalctl@ceph.mon.b.smithi055.stdout: block_restart_interval: 16 2024-03-26T01:56:43.795 INFO:journalctl@ceph.mon.b.smithi055.stdout: index_block_restart_interval: 1 2024-03-26T01:56:43.795 INFO:journalctl@ceph.mon.b.smithi055.stdout: metadata_block_size: 4096 2024-03-26T01:56:43.795 INFO:journalctl@ceph.mon.b.smithi055.stdout: partition_filters: 0 2024-03-26T01:56:43.796 INFO:journalctl@ceph.mon.b.smithi055.stdout: use_delta_encoding: 1 2024-03-26T01:56:43.796 INFO:journalctl@ceph.mon.b.smithi055.stdout: filter_policy: bloomfilter 2024-03-26T01:56:43.796 INFO:journalctl@ceph.mon.b.smithi055.stdout: whole_key_filtering: 1 2024-03-26T01:56:43.796 INFO:journalctl@ceph.mon.b.smithi055.stdout: verify_compression: 0 2024-03-26T01:56:43.796 INFO:journalctl@ceph.mon.b.smithi055.stdout: read_amp_bytes_per_bit: 0 2024-03-26T01:56:43.796 INFO:journalctl@ceph.mon.b.smithi055.stdout: format_version: 5 2024-03-26T01:56:43.797 INFO:journalctl@ceph.mon.b.smithi055.stdout: enable_index_compression: 1 2024-03-26T01:56:43.797 INFO:journalctl@ceph.mon.b.smithi055.stdout: block_align: 0 2024-03-26T01:56:43.797 INFO:journalctl@ceph.mon.b.smithi055.stdout: max_auto_readahead_size: 262144 2024-03-26T01:56:43.797 INFO:journalctl@ceph.mon.b.smithi055.stdout: prepopulate_block_cache: 0 2024-03-26T01:56:43.797 INFO:journalctl@ceph.mon.b.smithi055.stdout: initial_auto_readahead_size: 8192 2024-03-26T01:56:43.797 INFO:journalctl@ceph.mon.b.smithi055.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T01:56:43.798 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T01:56:43.798 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T01:56:43.798 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compression: NoCompression 2024-03-26T01:56:43.798 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T01:56:43.798 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T01:56:43.798 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T01:56:43.798 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.num_levels: 7 2024-03-26T01:56:43.799 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T01:56:43.799 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T01:56:43.799 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T01:56:43.799 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T01:56:43.799 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T01:56:43.799 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T01:56:43.799 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T01:56:43.800 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T01:56:43.800 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T01:56:43.800 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T01:56:43.800 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T01:56:43.800 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T01:56:43.800 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T01:56:43.801 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T01:56:43.801 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T01:56:43.801 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T01:56:43.801 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T01:56:43.801 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T01:56:43.801 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T01:56:43.801 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T01:56:43.802 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T01:56:43.802 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T01:56:43.802 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T01:56:43.802 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T01:56:43.802 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T01:56:43.802 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T01:56:43.802 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T01:56:43.803 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T01:56:43.803 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T01:56:43.803 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T01:56:43.803 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T01:56:43.805 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T01:56:43.805 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T01:56:43.805 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T01:56:43.806 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T01:56:43.806 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T01:56:43.806 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T01:56:43.806 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T01:56:43.806 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T01:56:43.806 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T01:56:43.807 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T01:56:43.807 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T01:56:43.807 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T01:56:43.807 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T01:56:43.807 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T01:56:43.807 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T01:56:43.807 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T01:56:43.808 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T01:56:43.808 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T01:56:43.808 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T01:56:43.808 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T01:56:43.808 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T01:56:43.808 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T01:56:43.808 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.table_properties_collectors: 2024-03-26T01:56:43.809 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.inplace_update_support: 0 2024-03-26T01:56:43.809 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T01:56:43.809 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T01:56:43.809 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T01:56:43.809 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T01:56:43.809 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.bloom_locality: 0 2024-03-26T01:56:43.809 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.max_successive_merges: 0 2024-03-26T01:56:43.810 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T01:56:43.810 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T01:56:43.810 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T01:56:43.810 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T01:56:43.810 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.ttl: 2592000 2024-03-26T01:56:43.810 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T01:56:43.810 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T01:56:43.811 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T01:56:43.811 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.enable_blob_files: false 2024-03-26T01:56:43.811 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.min_blob_size: 0 2024-03-26T01:56:43.811 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T01:56:43.811 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T01:56:43.811 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T01:56:43.811 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T01:56:43.812 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T01:56:43.812 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T01:56:43.812 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T01:56:43.812 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T01:56:43.812 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T01:56:43.812 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-26T01:56:43.812 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e463db7d-5e5d-4a47-9472-be1b534cb993 2024-03-26T01:56:43.813 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418203554072, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-26T01:56:43.813 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-26T01:56:43.813 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418203554704, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711418203, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e463db7d-5e5d-4a47-9472-be1b534cb993", "db_session_id": "HGPP249KPP2P2JBH56FX", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-26T01:56:43.813 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418203554802, "job": 1, "event": "recovery_finished"} 2024-03-26T01:56:43.813 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-26T01:56:43.813 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T01:56:43.813 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55600fe0e000 2024-03-26T01:56:43.813 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: DB pointer 0x55600fdf8000 2024-03-26T01:56:43.814 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T01:56:43.814 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T01:56:43.814 INFO:journalctl@ceph.mon.b.smithi055.stdout: ** DB Stats ** 2024-03-26T01:56:43.814 INFO:journalctl@ceph.mon.b.smithi055.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T01:56:43.814 INFO:journalctl@ceph.mon.b.smithi055.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T01:56:43.814 INFO:journalctl@ceph.mon.b.smithi055.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T01:56:43.815 INFO:journalctl@ceph.mon.b.smithi055.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T01:56:43.815 INFO:journalctl@ceph.mon.b.smithi055.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T01:56:43.815 INFO:journalctl@ceph.mon.b.smithi055.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T01:56:43.815 INFO:journalctl@ceph.mon.b.smithi055.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T01:56:43.815 INFO:journalctl@ceph.mon.b.smithi055.stdout: 2024-03-26T01:56:43.815 INFO:journalctl@ceph.mon.b.smithi055.stdout: ** Compaction Stats [default] ** 2024-03-26T01:56:43.815 INFO:journalctl@ceph.mon.b.smithi055.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T01:56:43.816 INFO:journalctl@ceph.mon.b.smithi055.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T01:56:43.816 INFO:journalctl@ceph.mon.b.smithi055.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T01:56:43.816 INFO:journalctl@ceph.mon.b.smithi055.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T01:56:43.816 INFO:journalctl@ceph.mon.b.smithi055.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T01:56:43.816 INFO:journalctl@ceph.mon.b.smithi055.stdout: 2024-03-26T01:56:43.816 INFO:journalctl@ceph.mon.b.smithi055.stdout: ** Compaction Stats [default] ** 2024-03-26T01:56:43.816 INFO:journalctl@ceph.mon.b.smithi055.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T01:56:43.817 INFO:journalctl@ceph.mon.b.smithi055.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T01:56:43.817 INFO:journalctl@ceph.mon.b.smithi055.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T01:56:43.817 INFO:journalctl@ceph.mon.b.smithi055.stdout: 2024-03-26T01:56:43.817 INFO:journalctl@ceph.mon.b.smithi055.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T01:56:43.817 INFO:journalctl@ceph.mon.b.smithi055.stdout: 2024-03-26T01:56:43.817 INFO:journalctl@ceph.mon.b.smithi055.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T01:56:43.817 INFO:journalctl@ceph.mon.b.smithi055.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T01:56:43.818 INFO:journalctl@ceph.mon.b.smithi055.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T01:56:43.818 INFO:journalctl@ceph.mon.b.smithi055.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T01:56:43.818 INFO:journalctl@ceph.mon.b.smithi055.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T01:56:43.818 INFO:journalctl@ceph.mon.b.smithi055.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T01:56:43.818 INFO:journalctl@ceph.mon.b.smithi055.stdout: Cumulative compaction: 0.00 GB write, 0.31 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T01:56:43.818 INFO:journalctl@ceph.mon.b.smithi055.stdout: Interval compaction: 0.00 GB write, 0.31 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T01:56:43.818 INFO:journalctl@ceph.mon.b.smithi055.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T01:56:43.819 INFO:journalctl@ceph.mon.b.smithi055.stdout: Block cache BinnedLRUCache@0x55600fd1d090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-26T01:56:43.819 INFO:journalctl@ceph.mon.b.smithi055.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-26T01:56:43.819 INFO:journalctl@ceph.mon.b.smithi055.stdout: 2024-03-26T01:56:43.819 INFO:journalctl@ceph.mon.b.smithi055.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T01:56:43.819 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-26T01:56:43.819 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: using public_addr v2:172.21.15.55:0/0 -> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] 2024-03-26T01:56:43.819 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: starting mon.b rank -1 at public addrs [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] at bind addrs [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T01:56:43.820 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: mon.b@-1(???) e0 preinit fsid b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T01:56:43.820 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: mon.b@-1(synchronizing).mds e1 new map 2024-03-26T01:56:43.820 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-26T01:56:43.820 INFO:journalctl@ceph.mon.b.smithi055.stdout: e1 2024-03-26T01:56:43.820 INFO:journalctl@ceph.mon.b.smithi055.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T01:56:43.820 INFO:journalctl@ceph.mon.b.smithi055.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T01:56:43.820 INFO:journalctl@ceph.mon.b.smithi055.stdout: legacy client fscid: -1 2024-03-26T01:56:43.820 INFO:journalctl@ceph.mon.b.smithi055.stdout: 2024-03-26T01:56:43.821 INFO:journalctl@ceph.mon.b.smithi055.stdout: No filesystems configured 2024-03-26T01:56:43.821 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-26T01:56:43.821 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-26T01:56:43.821 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-26T01:56:43.821 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-26T01:56:43.821 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-26T01:56:43.822 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-26T01:56:43.822 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T01:56:43.822 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T01:56:43.822 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T01:56:43.822 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T01:56:43.822 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:56:43.822 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:43.823 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:43.823 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi066", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:43.823 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: Deploying cephadm binary to smithi066 2024-03-26T01:56:43.823 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.823 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.823 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.823 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.824 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi055", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:56:43.824 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:43.824 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:56:43.824 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: Updating smithi055:/etc/ceph/ceph.conf 2024-03-26T01:56:43.824 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: Updating smithi055:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:43.824 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:56:43.824 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:43.825 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: Updating smithi055:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:56:43.825 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.825 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.825 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.825 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.825 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:43.825 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:43.826 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: Added host smithi066 2024-03-26T01:56:43.826 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.826 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.826 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:43.826 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.826 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:56:43.826 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:43.827 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/2525554678' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T01:56:43.827 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.827 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.827 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.827 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.827 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi066", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:56:43.827 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:43.828 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:56:43.828 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:43.828 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-26T01:56:43.828 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/2525554678' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T01:56:43.828 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:56:43.828 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.828 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.829 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.829 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: Updating smithi066:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:43.829 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:56:43.829 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: Updating smithi066:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:56:43.829 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:43.829 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.829 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:43.829 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:43.830 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:56:43.830 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.830 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:56:43.830 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:43.830 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi003:172.21.15.3=a;smithi055:172.21.15.55=b;smithi066:172.21.15.66=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:43.830 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: Saving service mon spec with placement smithi003:172.21.15.3=a;smithi055:172.21.15.55=b;smithi066:172.21.15.66=c;count:3 2024-03-26T01:56:43.830 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: Deploying daemon mon.c on smithi066 2024-03-26T01:56:43.831 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:43.831 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:43.831 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:43.831 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:56:43.831 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: mon.a calling monitor election 2024-03-26T01:56:43.831 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:43.831 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='client.? 172.21.15.66:0/1440825932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:56:43.832 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:43.832 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:43.832 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:43.832 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: mon.c calling monitor election 2024-03-26T01:56:43.832 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:43.832 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:43.832 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:43.833 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:43.833 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T01:56:43.833 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: monmap e2: 2 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],c=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:56:43.833 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: fsmap 2024-03-26T01:56:43.833 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:56:43.833 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: mgrmap e13: y(active, since 51s) 2024-03-26T01:56:43.833 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: overall HEALTH_OK 2024-03-26T01:56:43.834 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.834 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.834 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:43.834 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:56:43.834 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:43.834 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: Deploying daemon mon.b on smithi055 2024-03-26T01:56:43.834 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:43.835 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:43.835 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 ceph-mon[30619]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-26T01:56:43.835 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 bash[30580]: 6085b4c176b300748781c9cf6b2fced119c6f9d35fe10a0b18e83cb6b0a61e19 2024-03-26T01:56:43.835 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:43 smithi055 systemd[1]: Started Ceph mon.b for b95eae18-eb13-11ee-95d0-87774f69a715. 2024-03-26T01:56:44.995 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T01:56:44.996 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T01:56:45.253 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T01:56:48.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:56:48.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: mon.a calling monitor election 2024-03-26T01:56:48.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: mon.c calling monitor election 2024-03-26T01:56:48.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:48.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:48.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:48.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:48.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:48.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:48.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:48.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:48.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:48.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T01:56:48.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0],c=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:56:48.979 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: fsmap 2024-03-26T01:56:48.979 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:56:48.979 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: mgrmap e13: y(active, since 60s) 2024-03-26T01:56:48.979 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T01:56:48.979 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T01:56:48.979 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T01:56:48.979 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: mon.b (rank 2) addr [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] is down (out of quorum) 2024-03-26T01:56:48.979 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:48.979 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:48.979 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:48 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:49.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:56:49.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: mon.a calling monitor election 2024-03-26T01:56:49.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: mon.c calling monitor election 2024-03-26T01:56:49.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:49.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:49.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:49.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:49.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:49.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:49.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:49.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:49.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:49.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T01:56:49.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0],c=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:56:49.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: fsmap 2024-03-26T01:56:49.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:56:49.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: mgrmap e13: y(active, since 60s) 2024-03-26T01:56:49.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T01:56:49.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T01:56:49.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T01:56:49.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: mon.b (rank 2) addr [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] is down (out of quorum) 2024-03-26T01:56:49.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:49.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:49.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:48 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:49.932 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T01:56:49.932 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":3,"fsid":"b95eae18-eb13-11ee-95d0-87774f69a715","modified":"2024-03-26T01:56:43.583649Z","created":"2024-03-26T01:54:44.578800Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:3300","nonce":0},{"type":"v1","addr":"172.21.15.66:6789","nonce":0}]},"addr":"172.21.15.66:6789/0","public_addr":"172.21.15.66:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T01:56:49.932 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:49 smithi066 ceph-mon[30259]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:49.932 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:49 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:49.932 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:49 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:49.932 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:49 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:49.934 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 3 2024-03-26T01:56:50.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:49 smithi003 ceph-mon[25810]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:50.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:49 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:50.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:49 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:50.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:49 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:50.760 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-26T01:56:50.761 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-26T01:56:51.002 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:56:51.019 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:56:51.019 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: mon.a calling monitor election 2024-03-26T01:56:51.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: mon.c calling monitor election 2024-03-26T01:56:51.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:51.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:56:51.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:51.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:51.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:51.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:51.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:51.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:51.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:51.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T01:56:51.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0],c=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:56:51.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: fsmap 2024-03-26T01:56:51.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:56:51.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: mgrmap e13: y(active, since 60s) 2024-03-26T01:56:51.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T01:56:51.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T01:56:51.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T01:56:51.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: mon.b (rank 2) addr [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] is down (out of quorum) 2024-03-26T01:56:51.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:51.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:51.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:50 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:51.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: mon.b calling monitor election 2024-03-26T01:56:51.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:51.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: Updating smithi003:/etc/ceph/ceph.conf 2024-03-26T01:56:51.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: Updating smithi055:/etc/ceph/ceph.conf 2024-03-26T01:56:51.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-26T01:56:51.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: Updating smithi003:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:51.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: Updating smithi066:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:51.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: Updating smithi055:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:51.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: mon.b calling monitor election 2024-03-26T01:56:51.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: mon.c calling monitor election 2024-03-26T01:56:51.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: mon.a calling monitor election 2024-03-26T01:56:51.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T01:56:51.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0],c=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:56:51.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: fsmap 2024-03-26T01:56:51.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:56:51.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: mgrmap e13: y(active, since 62s) 2024-03-26T01:56:51.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-26T01:56:51.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: Cluster is now healthy 2024-03-26T01:56:51.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: overall HEALTH_OK 2024-03-26T01:56:51.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.393 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.393 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.393 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.393 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.393 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.393 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.393 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.393 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.393 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.393 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:56:51.393 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:56:51.393 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:51.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: mon.b calling monitor election 2024-03-26T01:56:51.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:51.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: Updating smithi003:/etc/ceph/ceph.conf 2024-03-26T01:56:51.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: Updating smithi055:/etc/ceph/ceph.conf 2024-03-26T01:56:51.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-26T01:56:51.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: Updating smithi003:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:51.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: Updating smithi066:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:51.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: Updating smithi055:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:51.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: mon.b calling monitor election 2024-03-26T01:56:51.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: mon.c calling monitor election 2024-03-26T01:56:51.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: mon.a calling monitor election 2024-03-26T01:56:51.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T01:56:51.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0],c=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:56:51.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: fsmap 2024-03-26T01:56:51.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:56:51.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: mgrmap e13: y(active, since 62s) 2024-03-26T01:56:51.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-26T01:56:51.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: Cluster is now healthy 2024-03-26T01:56:51.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: overall HEALTH_OK 2024-03-26T01:56:51.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.730 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.730 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.730 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.730 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:56:51.730 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:56:51.730 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:51 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:51.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: mon.b calling monitor election 2024-03-26T01:56:51.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:51.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: Updating smithi003:/etc/ceph/ceph.conf 2024-03-26T01:56:51.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: Updating smithi055:/etc/ceph/ceph.conf 2024-03-26T01:56:51.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-26T01:56:51.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: Updating smithi003:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:51.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: Updating smithi066:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:51.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: Updating smithi055:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:51.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: mon.b calling monitor election 2024-03-26T01:56:51.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: mon.c calling monitor election 2024-03-26T01:56:51.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: mon.a calling monitor election 2024-03-26T01:56:51.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T01:56:51.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0],c=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:56:51.771 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: fsmap 2024-03-26T01:56:51.771 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:56:51.771 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: mgrmap e13: y(active, since 62s) 2024-03-26T01:56:51.771 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-26T01:56:51.771 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: Cluster is now healthy 2024-03-26T01:56:51.771 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: overall HEALTH_OK 2024-03-26T01:56:51.771 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.771 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.771 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.771 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.771 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.771 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.771 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.771 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.772 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.772 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.772 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:51.772 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:56:51.772 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:56:51.772 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:51 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:52.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:52 smithi003 ceph-mon[25810]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T01:56:52.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:52 smithi003 ceph-mon[25810]: Reconfiguring daemon mon.a on smithi003 2024-03-26T01:56:52.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:52 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:52.663 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:56:52 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:56:52.584+0000 7f6b45760700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-26T01:56:52.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:52 smithi066 ceph-mon[30259]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T01:56:52.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:52 smithi066 ceph-mon[30259]: Reconfiguring daemon mon.a on smithi003 2024-03-26T01:56:52.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:52 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:52.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:52 smithi055 ceph-mon[30619]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T01:56:52.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:52 smithi055 ceph-mon[30619]: Reconfiguring daemon mon.a on smithi003 2024-03-26T01:56:52.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:52 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:56:53.074 INFO:teuthology.orchestra.run.smithi003.stdout:# minimal ceph.conf for b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T01:56:53.074 INFO:teuthology.orchestra.run.smithi003.stdout:[global] 2024-03-26T01:56:53.074 INFO:teuthology.orchestra.run.smithi003.stdout: fsid = b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T01:56:53.074 INFO:teuthology.orchestra.run.smithi003.stdout: mon_host = [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] 2024-03-26T01:56:53.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:53 smithi003 ceph-mon[25810]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:53.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:53 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:53.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:53 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/3358430166' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:53.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:53 smithi066 ceph-mon[30259]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:53.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:53 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:53.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:53 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/3358430166' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:53.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:53 smithi055 ceph-mon[30619]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:53.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:53 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:53.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:53 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/3358430166' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:54.306 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-26T01:56:54.307 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-26T01:56:54.307 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T01:56:54.339 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-26T01:56:54.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:56:54.409 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T01:56:54.409 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T01:56:54.441 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T01:56:54.441 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:56:54.508 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T01:56:54.508 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T01:56:54.547 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T01:56:54.547 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:56:54.622 INFO:tasks.cephadm:Adding mgr.y on smithi003 2024-03-26T01:56:54.622 INFO:tasks.cephadm:Adding mgr.x on smithi055 2024-03-26T01:56:54.622 INFO:tasks.cephadm:Adding mgr.z on smithi066 2024-03-26T01:56:54.622 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph orch apply mgr '3;smithi003=y;smithi055=x;smithi066=z' 2024-03-26T01:56:54.907 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T01:56:55.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:54 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:55.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:54 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:55.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:54 smithi003 ceph-mon[25810]: Reconfiguring mon.b (monmap changed)... 2024-03-26T01:56:55.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:54 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:56:55.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:54 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:56:55.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:54 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:55.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:54 smithi003 ceph-mon[25810]: Reconfiguring daemon mon.b on smithi055 2024-03-26T01:56:55.142 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:54 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:55.143 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:54 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:55.143 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:54 smithi066 ceph-mon[30259]: Reconfiguring mon.b (monmap changed)... 2024-03-26T01:56:55.143 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:54 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:56:55.143 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:54 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:56:55.143 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:54 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:55.143 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:54 smithi066 ceph-mon[30259]: Reconfiguring daemon mon.b on smithi055 2024-03-26T01:56:55.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:54 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:55.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:54 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:55.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:54 smithi055 ceph-mon[30619]: Reconfiguring mon.b (monmap changed)... 2024-03-26T01:56:55.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:54 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:56:55.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:54 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:56:55.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:54 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:55.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:54 smithi055 ceph-mon[30619]: Reconfiguring daemon mon.b on smithi055 2024-03-26T01:56:56.155 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:55 smithi066 ceph-mon[30259]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:56.155 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:55 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:56.155 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:55 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:56.155 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:55 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:56:56.155 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:55 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:56:56.155 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:55 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:56.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:55 smithi055 ceph-mon[30619]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:56.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:55 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:56.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:55 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:56.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:55 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:56:56.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:55 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:56:56.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:55 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:56.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:55 smithi003 ceph-mon[25810]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:56.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:55 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:56.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:55 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:56.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:55 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:56:56.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:55 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:56:56.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:55 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:56.596 INFO:teuthology.orchestra.run.smithi066.stdout:Scheduled mgr update... 2024-03-26T01:56:57.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:56 smithi066 ceph-mon[30259]: Reconfiguring mon.c (monmap changed)... 2024-03-26T01:56:57.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:56 smithi066 ceph-mon[30259]: Reconfiguring daemon mon.c on smithi066 2024-03-26T01:56:57.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:56 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:57.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:56 smithi055 ceph-mon[30619]: Reconfiguring mon.c (monmap changed)... 2024-03-26T01:56:57.271 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:56 smithi055 ceph-mon[30619]: Reconfiguring daemon mon.c on smithi066 2024-03-26T01:56:57.271 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:56 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:57.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:56 smithi003 ceph-mon[25810]: Reconfiguring mon.c (monmap changed)... 2024-03-26T01:56:57.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:56 smithi003 ceph-mon[25810]: Reconfiguring daemon mon.c on smithi066 2024-03-26T01:56:57.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:56 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:57.542 DEBUG:teuthology.orchestra.run.smithi055:mgr.x> sudo journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mgr.x.service 2024-03-26T01:56:57.547 DEBUG:teuthology.orchestra.run.smithi066:mgr.z> sudo journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mgr.z.service 2024-03-26T01:56:57.551 INFO:tasks.cephadm:Deploying OSDs... 2024-03-26T01:56:57.551 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-26T01:56:57.551 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T01:56:57.571 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T01:56:57.571 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/nvme1n1 2024-03-26T01:56:57.631 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/nvme1n1 2024-03-26T01:56:57.632 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T01:56:57.632 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-26T01:56:57.632 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T01:56:57.632 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T01:56:57.632 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-26 01:55:58.869951791 +0000 2024-03-26T01:56:57.632 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-26 01:54:00.183425488 +0000 2024-03-26T01:56:57.632 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-26 01:54:00.183425488 +0000 2024-03-26T01:56:57.632 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-26 01:54:00.172425800 +0000 2024-03-26T01:56:57.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-26T01:56:57.703 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-26T01:56:57.704 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-26T01:56:57.704 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000246962 s, 2.1 MB/s 2024-03-26T01:56:57.705 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-26T01:56:57.763 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/nvme2n1 2024-03-26T01:56:57.819 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/nvme2n1 2024-03-26T01:56:57.819 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T01:56:57.819 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-26T01:56:57.819 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T01:56:57.820 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T01:56:57.820 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-26 01:55:58.880951655 +0000 2024-03-26T01:56:57.820 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-26 01:54:00.338421094 +0000 2024-03-26T01:56:57.820 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-26 01:54:00.338421094 +0000 2024-03-26T01:56:57.820 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-26 01:54:00.329421349 +0000 2024-03-26T01:56:57.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-26T01:56:57.887 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-26T01:56:57.887 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-26T01:56:57.888 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000237102 s, 2.2 MB/s 2024-03-26T01:56:57.889 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-26T01:56:57.946 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/nvme3n1 2024-03-26T01:56:58.004 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/nvme3n1 2024-03-26T01:56:58.004 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T01:56:58.004 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-26T01:56:58.004 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T01:56:58.004 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T01:56:58.004 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-26 01:55:58.891951519 +0000 2024-03-26T01:56:58.004 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-26 01:54:00.493416701 +0000 2024-03-26T01:56:58.004 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-26 01:54:00.493416701 +0000 2024-03-26T01:56:58.004 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-26 01:54:00.484416956 +0000 2024-03-26T01:56:58.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-26T01:56:58.070 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:57 smithi003 ceph-mon[25810]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:58.070 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:57 smithi003 ceph-mon[25810]: from='client.24101 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi003=y;smithi055=x;smithi066=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:58.070 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:57 smithi003 ceph-mon[25810]: Saving service mgr spec with placement smithi003=y;smithi055=x;smithi066=z;count:3 2024-03-26T01:56:58.072 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-26T01:56:58.072 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-26T01:56:58.073 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000211079 s, 2.4 MB/s 2024-03-26T01:56:58.074 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-26T01:56:58.130 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/nvme4n1 2024-03-26T01:56:58.185 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/nvme4n1 2024-03-26T01:56:58.185 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T01:56:58.185 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-26T01:56:58.186 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T01:56:58.186 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T01:56:58.186 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-26 01:55:58.902951384 +0000 2024-03-26T01:56:58.186 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-26 01:54:00.648412307 +0000 2024-03-26T01:56:58.186 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-26 01:54:00.648412307 +0000 2024-03-26T01:56:58.186 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-26 01:54:00.636412647 +0000 2024-03-26T01:56:58.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-26T01:56:58.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:57 smithi066 ceph-mon[30259]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:58.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:57 smithi066 ceph-mon[30259]: from='client.24101 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi003=y;smithi055=x;smithi066=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:58.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:57 smithi066 ceph-mon[30259]: Saving service mgr spec with placement smithi003=y;smithi055=x;smithi066=z;count:3 2024-03-26T01:56:58.251 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-26T01:56:58.251 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-26T01:56:58.251 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000247063 s, 2.1 MB/s 2024-03-26T01:56:58.252 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-26T01:56:58.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:57 smithi055 ceph-mon[30619]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:58.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:57 smithi055 ceph-mon[30619]: from='client.24101 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi003=y;smithi055=x;smithi066=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:58.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:57 smithi055 ceph-mon[30619]: Saving service mgr spec with placement smithi003=y;smithi055=x;smithi066=z;count:3 2024-03-26T01:56:58.307 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T01:56:58.308 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T01:56:58.326 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T01:56:58.326 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/nvme1n1 2024-03-26T01:56:58.383 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/nvme1n1 2024-03-26T01:56:58.383 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T01:56:58.383 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-26T01:56:58.383 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T01:56:58.383 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T01:56:58.384 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-26 01:56:15.875614801 +0000 2024-03-26T01:56:58.384 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-26 01:54:01.841349736 +0000 2024-03-26T01:56:58.384 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-26 01:54:01.841349736 +0000 2024-03-26T01:56:58.384 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-03-26 01:54:01.828350001 +0000 2024-03-26T01:56:58.384 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-26T01:56:58.453 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-26T01:56:58.454 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-26T01:56:58.454 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000318958 s, 1.6 MB/s 2024-03-26T01:56:58.455 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-26T01:56:58.513 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/nvme2n1 2024-03-26T01:56:58.570 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/nvme2n1 2024-03-26T01:56:58.570 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T01:56:58.570 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-26T01:56:58.570 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T01:56:58.571 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T01:56:58.571 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-26 01:56:15.887614558 +0000 2024-03-26T01:56:58.571 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-26 01:54:01.984346815 +0000 2024-03-26T01:56:58.571 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-26 01:54:01.984346815 +0000 2024-03-26T01:56:58.571 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-03-26 01:54:01.972347060 +0000 2024-03-26T01:56:58.571 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-26T01:56:58.638 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-26T01:56:58.638 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-26T01:56:58.638 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000320355 s, 1.6 MB/s 2024-03-26T01:56:58.639 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-26T01:56:58.696 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/nvme3n1 2024-03-26T01:56:58.752 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/nvme3n1 2024-03-26T01:56:58.753 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T01:56:58.753 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 786 Links: 1 Device type: 103,6 2024-03-26T01:56:58.753 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T01:56:58.753 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T01:56:58.753 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-26 01:56:15.899614315 +0000 2024-03-26T01:56:58.754 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-26 01:54:02.137343691 +0000 2024-03-26T01:56:58.754 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-26 01:54:02.137343691 +0000 2024-03-26T01:56:58.754 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-03-26 01:54:02.123343976 +0000 2024-03-26T01:56:58.754 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-26T01:56:58.819 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-26T01:56:58.820 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-26T01:56:58.820 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000293228 s, 1.7 MB/s 2024-03-26T01:56:58.821 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-26T01:56:58.876 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/nvme4n1 2024-03-26T01:56:58.932 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/nvme4n1 2024-03-26T01:56:58.932 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T01:56:58.932 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 798 Links: 1 Device type: 103,8 2024-03-26T01:56:58.932 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T01:56:58.932 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T01:56:58.932 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-26 01:56:15.911614071 +0000 2024-03-26T01:56:58.932 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-26 01:54:02.247341444 +0000 2024-03-26T01:56:58.932 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-26 01:54:02.247341444 +0000 2024-03-26T01:56:58.932 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-03-26 01:54:02.229341812 +0000 2024-03-26T01:56:58.932 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-26T01:56:59.000 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-26T01:56:59.000 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-26T01:56:59.000 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.0017466 s, 293 kB/s 2024-03-26T01:56:59.001 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-26T01:56:59.057 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T01:56:59.057 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T01:56:59.073 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T01:56:59.073 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/nvme1n1 2024-03-26T01:56:59.131 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/nvme1n1 2024-03-26T01:56:59.131 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T01:56:59.131 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-26T01:56:59.131 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T01:56:59.132 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T01:56:59.132 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-26 01:56:25.766272782 +0000 2024-03-26T01:56:59.132 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-26 01:54:03.557554399 +0000 2024-03-26T01:56:59.132 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-26 01:54:03.557554399 +0000 2024-03-26T01:56:59.132 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-26 01:54:03.540554553 +0000 2024-03-26T01:56:59.133 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-26T01:56:59.204 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-26T01:56:59.204 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-26T01:56:59.204 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.00027147 s, 1.9 MB/s 2024-03-26T01:56:59.205 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-26T01:56:59.261 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/nvme2n1 2024-03-26T01:56:59.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:59 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:59.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:59 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:59.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:59 smithi055 ceph-mon[30619]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:59.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:59 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:59.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:59 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:59.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:59 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:56:59.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:59 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:59.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:59 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:56:59.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:59 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T01:56:59.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:59 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:56:59.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:59 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:59.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:56:59 smithi055 ceph-mon[30619]: Deploying daemon mgr.z on smithi066 2024-03-26T01:56:59.317 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/nvme2n1 2024-03-26T01:56:59.317 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T01:56:59.317 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-03-26T01:56:59.317 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T01:56:59.318 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T01:56:59.318 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-26 01:56:25.777272682 +0000 2024-03-26T01:56:59.318 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-26 01:54:03.685553246 +0000 2024-03-26T01:56:59.318 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-26 01:54:03.685553246 +0000 2024-03-26T01:56:59.318 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-26 01:54:03.673553354 +0000 2024-03-26T01:56:59.318 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-26T01:56:59.325 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:58 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:59.326 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:58 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:59.326 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:58 smithi066 ceph-mon[30259]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:59.326 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:58 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:59.326 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:58 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:59.326 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:58 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:56:59.326 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:58 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:59.326 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:58 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:56:59.326 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:58 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T01:56:59.326 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:58 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:56:59.326 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:58 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:59.326 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:56:58 smithi066 ceph-mon[30259]: Deploying daemon mgr.z on smithi066 2024-03-26T01:56:59.356 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-26T01:56:59.356 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-26T01:56:59.356 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000252076 s, 2.0 MB/s 2024-03-26T01:56:59.357 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-26T01:56:59.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:58 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:59.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:58 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:59.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:58 smithi003 ceph-mon[25810]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:59.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:58 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:59.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:58 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:59.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:58 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:56:59.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:58 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:56:59.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:58 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:56:59.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:58 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T01:56:59.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:58 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:56:59.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:58 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:59.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:56:58 smithi003 ceph-mon[25810]: Deploying daemon mgr.z on smithi066 2024-03-26T01:56:59.413 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/nvme3n1 2024-03-26T01:56:59.469 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/nvme3n1 2024-03-26T01:56:59.469 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T01:56:59.469 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-26T01:56:59.469 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T01:56:59.469 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T01:56:59.470 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-26 01:56:25.789272573 +0000 2024-03-26T01:56:59.470 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-26 01:54:03.813552093 +0000 2024-03-26T01:56:59.470 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-26 01:54:03.813552093 +0000 2024-03-26T01:56:59.470 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-26 01:54:03.801552201 +0000 2024-03-26T01:56:59.470 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-26T01:56:59.540 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-26T01:56:59.540 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-26T01:56:59.540 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000218148 s, 2.3 MB/s 2024-03-26T01:56:59.541 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-26T01:56:59.597 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/nvme4n1 2024-03-26T01:56:59.654 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/nvme4n1 2024-03-26T01:56:59.654 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T01:56:59.654 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-26T01:56:59.654 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T01:56:59.654 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T01:56:59.654 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-26 01:56:25.800272473 +0000 2024-03-26T01:56:59.654 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-26 01:54:03.947550886 +0000 2024-03-26T01:56:59.655 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-26 01:54:03.947550886 +0000 2024-03-26T01:56:59.655 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-26 01:54:03.941550940 +0000 2024-03-26T01:56:59.655 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-26T01:56:59.724 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-26T01:56:59.724 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-26T01:56:59.724 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.00019784 s, 2.6 MB/s 2024-03-26T01:56:59.726 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-26T01:56:59.783 INFO:tasks.cephadm:Deploying osd.0 on smithi003 with /dev/nvme4n1... 2024-03-26T01:56:59.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-26T01:57:00.030 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:57:00.958 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:00 smithi066 systemd[1]: Starting Ceph mgr.z for b95eae18-eb13-11ee-95d0-87774f69a715... 2024-03-26T01:57:01.137 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:01 smithi003 ceph-mon[25810]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:01.216 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:01 smithi066 ceph-mon[30259]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:01.217 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:00 smithi066 podman[32345]: 2024-03-26 01:57:00.86139524 +0000 UTC m=+0.018320251 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T01:57:01.217 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:00 smithi066 podman[32345]: 2024-03-26 01:57:00.959474928 +0000 UTC m=+0.116399935 container create c8a4bef62a3e5eba405f5f8414adbb002a10b32135e00d306a96a1b6e90c914c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-26T01:57:01.217 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:01 smithi066 podman[32345]: 2024-03-26 01:57:01.186733606 +0000 UTC m=+0.343658615 container init c8a4bef62a3e5eba405f5f8414adbb002a10b32135e00d306a96a1b6e90c914c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-26T01:57:01.217 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:01 smithi066 podman[32345]: 2024-03-26 01:57:01.192317912 +0000 UTC m=+0.349242923 container start c8a4bef62a3e5eba405f5f8414adbb002a10b32135e00d306a96a1b6e90c914c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-26T01:57:01.217 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:01 smithi066 ceph-mgr[32384]: -- 172.21.15.66:0/27524975 <== mon.1 v2:172.21.15.66:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5619c95481a0 con 0x5619c951f000 2024-03-26T01:57:01.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:01 smithi055 ceph-mon[30619]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:01.477 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:01 smithi066 bash[32345]: c8a4bef62a3e5eba405f5f8414adbb002a10b32135e00d306a96a1b6e90c914c 2024-03-26T01:57:01.477 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:01 smithi066 systemd[1]: Started Ceph mgr.z for b95eae18-eb13-11ee-95d0-87774f69a715. 2024-03-26T01:57:01.995 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping: /dev/nvme4n1 2024-03-26T01:57:01.995 INFO:teuthology.orchestra.run.smithi003.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-26T01:57:01.995 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-26T01:57:01.995 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10+0 records in 2024-03-26T01:57:01.995 INFO:teuthology.orchestra.run.smithi003.stderr:10+0 records out 2024-03-26T01:57:01.995 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121987 s, 860 MB/s 2024-03-26T01:57:01.995 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping successful for: 2024-03-26T01:57:02.227 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:01 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:01.751+0000 7f91638b4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T01:57:02.227 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:01 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:01.847+0000 7f91638b4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T01:57:02.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:02 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:02.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:02 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:02.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:02 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:02.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:02 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:57:02.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:02 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T01:57:02.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:02 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:57:02.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:02 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:02.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:02 smithi003 ceph-mon[25810]: Deploying daemon mgr.x on smithi055 2024-03-26T01:57:02.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi003:/dev/nvme4n1 2024-03-26T01:57:02.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:02 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:02.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:02 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:02.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:02 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:02.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:02 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:57:02.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:02 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T01:57:02.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:02 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:57:02.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:02 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:02.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:02 smithi066 ceph-mon[30259]: Deploying daemon mgr.x on smithi055 2024-03-26T01:57:02.728 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:02 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:02.514+0000 7f91638b4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T01:57:02.728 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:02 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:02.610+0000 7f91638b4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T01:57:02.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:02 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:02.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:02 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:02.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:02 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:02.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:02 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:57:02.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:02 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T01:57:02.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:02 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:57:02.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:02 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:02.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:02 smithi055 ceph-mon[30619]: Deploying daemon mgr.x on smithi055 2024-03-26T01:57:02.958 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:57:03.211 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:02 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:02.756+0000 7f91638b4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T01:57:03.211 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:02 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:02.854+0000 7f91638b4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T01:57:03.211 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:02 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:02.939+0000 7f91638b4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T01:57:03.211 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:03 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:03.210+0000 7f91638b4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T01:57:03.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:03 smithi066 ceph-mon[30259]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:03.477 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:03 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:03.322+0000 7f91638b4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T01:57:03.477 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:03 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:03.412+0000 7f91638b4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T01:57:03.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:03 smithi003 ceph-mon[25810]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:03.659 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:03 smithi055 ceph-mon[30619]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:03.659 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:03 smithi055 podman[32065]: 2024-03-26 01:57:03.547965805 +0000 UTC m=+0.017748224 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T01:57:03.660 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:03 smithi055 podman[32065]: 2024-03-26 01:57:03.659999715 +0000 UTC m=+0.129782124 container create 40a447c1012e483f9c6cee59dbf08ed6474ffaed96eb9a64c4ec372e2f64a74d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, io.buildah.version=1.33.5) 2024-03-26T01:57:03.970 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:03 smithi055 podman[32065]: 2024-03-26 01:57:03.862519552 +0000 UTC m=+0.332301961 container init 40a447c1012e483f9c6cee59dbf08ed6474ffaed96eb9a64c4ec372e2f64a74d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T01:57:03.970 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:03 smithi055 podman[32065]: 2024-03-26 01:57:03.867930007 +0000 UTC m=+0.337712419 container start 40a447c1012e483f9c6cee59dbf08ed6474ffaed96eb9a64c4ec372e2f64a74d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124) 2024-03-26T01:57:03.970 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:03 smithi055 ceph-mgr[32103]: -- 172.21.15.55:0/2383481048 <== mon.2 v2:172.21.15.55:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55803ce461a0 con 0x55803ce1c800 2024-03-26T01:57:03.977 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:03 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:03.509+0000 7f91638b4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T01:57:03.977 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:03 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:03.690+0000 7f91638b4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T01:57:04.269 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:03 smithi055 bash[32065]: 40a447c1012e483f9c6cee59dbf08ed6474ffaed96eb9a64c4ec372e2f64a74d 2024-03-26T01:57:04.269 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:03 smithi055 systemd[1]: Started Ceph mgr.x for b95eae18-eb13-11ee-95d0-87774f69a715. 2024-03-26T01:57:04.769 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:04.431+0000 7f3960c2e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T01:57:04.769 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:04.528+0000 7f3960c2e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T01:57:05.037 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:04 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:04.729+0000 7f91638b4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T01:57:05.038 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:04 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:04.874+0000 7f91638b4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T01:57:05.306 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:05 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:05.241+0000 7f3960c2e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T01:57:05.306 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:05 smithi055 ceph-mon[30619]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:05.306 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:05 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:05.306 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:05 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:05.306 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:05 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:05.307 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:05 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:05.307 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:05 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:05.307 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:05 smithi055 ceph-mon[30619]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:05.307 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:05 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:57:05.307 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:05 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:57:05.307 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:05 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:05.318 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:05 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:05.224+0000 7f91638b4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T01:57:05.318 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:05 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:05.316+0000 7f91638b4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T01:57:05.318 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:05 smithi066 ceph-mon[30259]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:05.319 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:05 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:05.319 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:05 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:05.319 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:05 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:05.319 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:05 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:05.319 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:05 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:05.319 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:05 smithi066 ceph-mon[30259]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:05.319 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:05 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:57:05.319 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:05 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:57:05.319 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:05 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:05.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:05 smithi003 ceph-mon[25810]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:05.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:05 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:05.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:05 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:05.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:05 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:05.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:05 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:05.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:05 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:05.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:05 smithi003 ceph-mon[25810]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:05.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:05 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:57:05.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:05 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:57:05.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:05 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:05.587 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:05 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:05.336+0000 7f3960c2e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T01:57:05.587 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:05 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:05.479+0000 7f3960c2e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T01:57:05.587 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:05 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:05.587+0000 7f3960c2e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T01:57:05.716 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:05 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:05.524+0000 7f91638b4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:57:05.716 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:05 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:05.715+0000 7f91638b4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T01:57:05.940 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:05 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:05.673+0000 7f3960c2e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T01:57:05.940 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:05 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:05.940+0000 7f3960c2e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T01:57:06.236 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:06 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:06.051+0000 7f3960c2e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T01:57:06.236 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:06 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:06.139+0000 7f3960c2e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T01:57:06.236 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:06 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:06.236+0000 7f3960c2e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T01:57:06.275 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:06 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:06.006+0000 7f91638b4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:57:06.275 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:06 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:06.274+0000 7f91638b4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T01:57:06.519 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:06 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:06.413+0000 7f3960c2e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T01:57:06.554 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:06 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:06.518+0000 7f91638b4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T01:57:06.573 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:06 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:06.574 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:06 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:06.574 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:06 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:06.574 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:06 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:06.574 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:06 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:06.574 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:06 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:06.574 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:06 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:06.574 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:06 smithi003 ceph-mon[25810]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T01:57:06.574 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:06 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:57:06.574 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:06 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:57:06.574 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:06 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:06.574 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:06 smithi003 ceph-mon[25810]: Reconfiguring daemon mgr.y on smithi003 2024-03-26T01:57:06.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:06 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:06.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:06 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:06.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:06 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:06.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:06 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:06.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:06 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:06.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:06 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:06.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:06 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:06.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:06 smithi066 ceph-mon[30259]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T01:57:06.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:06 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:57:06.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:06 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:57:06.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:06 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:06.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:06 smithi066 ceph-mon[30259]: Reconfiguring daemon mgr.y on smithi003 2024-03-26T01:57:07.019 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:06 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:07.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:06 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:07.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:06 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:07.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:06 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:07.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:06 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:07.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:06 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:07.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:06 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:07.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:06 smithi055 ceph-mon[30619]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T01:57:07.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:06 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:57:07.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:06 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:57:07.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:06 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:07.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:06 smithi055 ceph-mon[30619]: Reconfiguring daemon mgr.y on smithi003 2024-03-26T01:57:07.615 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:07 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:07.348+0000 7f91638b4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T01:57:07.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:07 smithi055 ceph-mon[30619]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:07.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:07 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/782822368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b217e513-44cb-41f3-a109-5a3688f2c8ef"}]: dispatch 2024-03-26T01:57:07.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:07 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/782822368' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b217e513-44cb-41f3-a109-5a3688f2c8ef"}]': finished 2024-03-26T01:57:07.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:07 smithi055 ceph-mon[30619]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T01:57:07.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:07 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:07.770 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:07 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:07.411+0000 7f3960c2e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T01:57:07.770 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:07 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:07.544+0000 7f3960c2e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T01:57:07.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:07 smithi003 ceph-mon[25810]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:07.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:07 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/782822368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b217e513-44cb-41f3-a109-5a3688f2c8ef"}]: dispatch 2024-03-26T01:57:07.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:07 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/782822368' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b217e513-44cb-41f3-a109-5a3688f2c8ef"}]': finished 2024-03-26T01:57:07.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:07 smithi003 ceph-mon[25810]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T01:57:07.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:07 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:07.938 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:07 smithi066 ceph-mon[30259]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:07.939 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:07 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/782822368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b217e513-44cb-41f3-a109-5a3688f2c8ef"}]: dispatch 2024-03-26T01:57:07.939 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:07 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/782822368' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b217e513-44cb-41f3-a109-5a3688f2c8ef"}]': finished 2024-03-26T01:57:07.939 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:07 smithi066 ceph-mon[30259]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T01:57:07.939 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:07 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:08.168 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:07 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:07.879+0000 7f3960c2e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T01:57:08.168 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:07 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:07.970+0000 7f3960c2e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T01:57:08.168 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:08 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:08.167+0000 7f3960c2e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:57:08.227 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:07 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:07.938+0000 7f91638b4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T01:57:08.227 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:08 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:08.038+0000 7f91638b4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T01:57:08.519 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:08 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:08.354+0000 7f3960c2e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T01:57:08.727 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:57:08 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:57:08.260+0000 7f91638b4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T01:57:08.900 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:08 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:08.640+0000 7f3960c2e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:57:08.900 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:08 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:08.900+0000 7f3960c2e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T01:57:09.269 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:09 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:09.139+0000 7f3960c2e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T01:57:09.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:09 smithi055 ceph-mon[30619]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:09.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:09 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:09.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:09 smithi055 ceph-mon[30619]: Standby manager daemon z started 2024-03-26T01:57:09.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:09 smithi055 ceph-mon[30619]: from='mgr.? 172.21.15.66:0/3967862405' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T01:57:09.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:09 smithi055 ceph-mon[30619]: from='mgr.? 172.21.15.66:0/3967862405' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:57:09.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:09 smithi055 ceph-mon[30619]: from='mgr.? 172.21.15.66:0/3967862405' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T01:57:09.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:09 smithi055 ceph-mon[30619]: from='mgr.? 172.21.15.66:0/3967862405' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:57:09.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:09 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/1083123336' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:57:09.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:09 smithi066 ceph-mon[30259]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:09.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:09 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:09.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:09 smithi066 ceph-mon[30259]: Standby manager daemon z started 2024-03-26T01:57:09.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:09 smithi066 ceph-mon[30259]: from='mgr.? 172.21.15.66:0/3967862405' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T01:57:09.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:09 smithi066 ceph-mon[30259]: from='mgr.? 172.21.15.66:0/3967862405' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:57:09.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:09 smithi066 ceph-mon[30259]: from='mgr.? 172.21.15.66:0/3967862405' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T01:57:09.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:09 smithi066 ceph-mon[30259]: from='mgr.? 172.21.15.66:0/3967862405' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:57:09.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:09 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/1083123336' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:57:09.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:09 smithi003 ceph-mon[25810]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:09.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:09 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:09.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:09 smithi003 ceph-mon[25810]: Standby manager daemon z started 2024-03-26T01:57:09.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:09 smithi003 ceph-mon[25810]: from='mgr.? 172.21.15.66:0/3967862405' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T01:57:09.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:09 smithi003 ceph-mon[25810]: from='mgr.? 172.21.15.66:0/3967862405' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:57:09.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:09 smithi003 ceph-mon[25810]: from='mgr.? 172.21.15.66:0/3967862405' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T01:57:09.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:09 smithi003 ceph-mon[25810]: from='mgr.? 172.21.15.66:0/3967862405' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:57:09.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:09 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/1083123336' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:57:10.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:10 smithi055 ceph-mon[30619]: mgrmap e14: y(active, since 81s), standbys: z 2024-03-26T01:57:10.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:10 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T01:57:10.270 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:09 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:09.958+0000 7f3960c2e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T01:57:10.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:10 smithi003 ceph-mon[25810]: mgrmap e14: y(active, since 81s), standbys: z 2024-03-26T01:57:10.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:10 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T01:57:10.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:10 smithi066 ceph-mon[30259]: mgrmap e14: y(active, since 81s), standbys: z 2024-03-26T01:57:10.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:10 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T01:57:10.848 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:10 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:10.533+0000 7f3960c2e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T01:57:10.848 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:10 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:10.631+0000 7f3960c2e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T01:57:10.849 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:57:10 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:57:10.848+0000 7f3960c2e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T01:57:11.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:11 smithi055 ceph-mon[30619]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:11.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:11 smithi055 ceph-mon[30619]: Standby manager daemon x started 2024-03-26T01:57:11.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:11 smithi055 ceph-mon[30619]: from='mgr.? 172.21.15.55:0/1998358534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T01:57:11.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:11 smithi055 ceph-mon[30619]: from='mgr.? 172.21.15.55:0/1998358534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:57:11.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:11 smithi055 ceph-mon[30619]: from='mgr.? 172.21.15.55:0/1998358534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T01:57:11.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:11 smithi055 ceph-mon[30619]: from='mgr.? 172.21.15.55:0/1998358534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:57:11.371 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:11 smithi003 ceph-mon[25810]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:11.371 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:11 smithi003 ceph-mon[25810]: Standby manager daemon x started 2024-03-26T01:57:11.371 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:11 smithi003 ceph-mon[25810]: from='mgr.? 172.21.15.55:0/1998358534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T01:57:11.371 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:11 smithi003 ceph-mon[25810]: from='mgr.? 172.21.15.55:0/1998358534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:57:11.371 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:11 smithi003 ceph-mon[25810]: from='mgr.? 172.21.15.55:0/1998358534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T01:57:11.371 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:11 smithi003 ceph-mon[25810]: from='mgr.? 172.21.15.55:0/1998358534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:57:11.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:11 smithi066 ceph-mon[30259]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:11.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:11 smithi066 ceph-mon[30259]: Standby manager daemon x started 2024-03-26T01:57:11.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:11 smithi066 ceph-mon[30259]: from='mgr.? 172.21.15.55:0/1998358534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T01:57:11.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:11 smithi066 ceph-mon[30259]: from='mgr.? 172.21.15.55:0/1998358534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:57:11.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:11 smithi066 ceph-mon[30259]: from='mgr.? 172.21.15.55:0/1998358534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T01:57:11.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:11 smithi066 ceph-mon[30259]: from='mgr.? 172.21.15.55:0/1998358534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:57:12.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:12 smithi003 ceph-mon[25810]: mgrmap e15: y(active, since 83s), standbys: z, x 2024-03-26T01:57:12.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:12 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T01:57:12.362 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:12 smithi055 ceph-mon[30619]: mgrmap e15: y(active, since 83s), standbys: z, x 2024-03-26T01:57:12.362 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:12 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T01:57:12.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:12 smithi066 ceph-mon[30259]: mgrmap e15: y(active, since 83s), standbys: z, x 2024-03-26T01:57:12.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:12 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T01:57:13.261 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:13 smithi003 ceph-mon[25810]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:13.262 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:13 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:13.262 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:13 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:13.262 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:13 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:13.262 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:13 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:13.262 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:13 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:13.262 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:13 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:13.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:13 smithi066 ceph-mon[30259]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:13.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:13 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:13.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:13 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:13.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:13 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:13.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:13 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:13.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:13 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:13.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:13 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:13.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:13 smithi055 ceph-mon[30619]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:13.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:13 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:13.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:13 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:13.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:13 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:13.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:13 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:13.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:13 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:13.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:13 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:15.281 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:15 smithi003 ceph-mon[25810]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:15.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:15 smithi066 ceph-mon[30259]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:15.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:15 smithi055 ceph-mon[30619]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:16.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:16 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T01:57:16.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:16 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:16.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:16 smithi003 ceph-mon[25810]: Deploying daemon osd.0 on smithi003 2024-03-26T01:57:16.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:16 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T01:57:16.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:16 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:16.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:16 smithi066 ceph-mon[30259]: Deploying daemon osd.0 on smithi003 2024-03-26T01:57:16.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:16 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T01:57:16.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:16 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:16.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:16 smithi055 ceph-mon[30619]: Deploying daemon osd.0 on smithi003 2024-03-26T01:57:17.212 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:17 smithi003 ceph-mon[25810]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:17.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:17 smithi066 ceph-mon[30259]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:17.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:17 smithi055 ceph-mon[30619]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:19.293 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:19 smithi003 ceph-mon[25810]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:19.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:19 smithi066 ceph-mon[30259]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:19.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:19 smithi055 ceph-mon[30619]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:21.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:21 smithi003 ceph-mon[25810]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:21.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:21 smithi066 ceph-mon[30259]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:21.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:21 smithi055 ceph-mon[30619]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:22.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:22 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:22.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:22 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:22.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:22 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:22.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:22 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:22.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:22 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:22.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:22 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:22.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:22 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:22.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:22 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:22.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:22 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:23.589 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:23 smithi003 ceph-mon[25810]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:23.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:23 smithi066 ceph-mon[30259]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:23.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:23 smithi055 ceph-mon[30619]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:24.888 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:24 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:24.888 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:24 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:24.888 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:24 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:24.888 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:24 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:24.888 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:24 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:24.888 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:24 smithi003 ceph-mon[25810]: from='osd.0 [v2:172.21.15.3:6802/388847396,v1:172.21.15.3:6803/388847396]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T01:57:25.019 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:24 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:25.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:24 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:25.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:24 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:25.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:24 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:25.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:24 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:25.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:24 smithi055 ceph-mon[30619]: from='osd.0 [v2:172.21.15.3:6802/388847396,v1:172.21.15.3:6803/388847396]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T01:57:25.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:24 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:25.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:24 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:25.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:24 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:25.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:24 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:25.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:24 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:25.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:24 smithi066 ceph-mon[30259]: from='osd.0 [v2:172.21.15.3:6802/388847396,v1:172.21.15.3:6803/388847396]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T01:57:25.268 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 0 on host 'smithi003' 2024-03-26T01:57:25.850 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:25 smithi003 ceph-mon[25810]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:25.850 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:25 smithi003 ceph-mon[25810]: from='osd.0 [v2:172.21.15.3:6802/388847396,v1:172.21.15.3:6803/388847396]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T01:57:25.850 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:25 smithi003 ceph-mon[25810]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T01:57:25.850 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:25 smithi003 ceph-mon[25810]: from='osd.0 [v2:172.21.15.3:6802/388847396,v1:172.21.15.3:6803/388847396]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-26T01:57:25.850 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:25 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:25.850 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:25 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:25.850 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:25 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:25.851 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:25 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:25.875 DEBUG:teuthology.orchestra.run.smithi003:osd.0> sudo journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@osd.0.service 2024-03-26T01:57:25.877 INFO:tasks.cephadm:Deploying osd.1 on smithi055 with /dev/nvme4n1... 2024-03-26T01:57:25.878 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-26T01:57:26.019 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:25 smithi055 ceph-mon[30619]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:26.019 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:25 smithi055 ceph-mon[30619]: from='osd.0 [v2:172.21.15.3:6802/388847396,v1:172.21.15.3:6803/388847396]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T01:57:26.019 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:25 smithi055 ceph-mon[30619]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T01:57:26.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:25 smithi055 ceph-mon[30619]: from='osd.0 [v2:172.21.15.3:6802/388847396,v1:172.21.15.3:6803/388847396]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-26T01:57:26.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:25 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:26.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:25 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:26.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:25 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:26.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:25 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:26.124 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T01:57:26.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:25 smithi066 ceph-mon[30259]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:26.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:25 smithi066 ceph-mon[30259]: from='osd.0 [v2:172.21.15.3:6802/388847396,v1:172.21.15.3:6803/388847396]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T01:57:26.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:25 smithi066 ceph-mon[30259]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T01:57:26.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:25 smithi066 ceph-mon[30259]: from='osd.0 [v2:172.21.15.3:6802/388847396,v1:172.21.15.3:6803/388847396]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-26T01:57:26.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:25 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:26.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:25 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:26.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:25 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:26.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:25 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:27.123 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:26 smithi003 ceph-mon[25810]: from='osd.0 [v2:172.21.15.3:6802/388847396,v1:172.21.15.3:6803/388847396]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-03-26T01:57:27.123 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:26 smithi003 ceph-mon[25810]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T01:57:27.123 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:26 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:27.123 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:26 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:27.123 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:26 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:27.220 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:26 smithi055 ceph-mon[30619]: from='osd.0 [v2:172.21.15.3:6802/388847396,v1:172.21.15.3:6803/388847396]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-03-26T01:57:27.220 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:26 smithi055 ceph-mon[30619]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T01:57:27.220 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:26 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:27.220 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:26 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:27.220 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:26 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:27.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:26 smithi066 ceph-mon[30259]: from='osd.0 [v2:172.21.15.3:6802/388847396,v1:172.21.15.3:6803/388847396]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-03-26T01:57:27.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:26 smithi066 ceph-mon[30259]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T01:57:27.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:26 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:27.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:26 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:27.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:26 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:28.014 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping: /dev/nvme4n1 2024-03-26T01:57:28.014 INFO:teuthology.orchestra.run.smithi055.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-26T01:57:28.014 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-26T01:57:28.014 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10+0 records in 2024-03-26T01:57:28.014 INFO:teuthology.orchestra.run.smithi055.stderr:10+0 records out 2024-03-26T01:57:28.014 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0155785 s, 673 MB/s 2024-03-26T01:57:28.014 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping successful for: 2024-03-26T01:57:28.037 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:27 smithi055 ceph-mon[30619]: purged_snaps scrub starts 2024-03-26T01:57:28.037 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:27 smithi055 ceph-mon[30619]: purged_snaps scrub ok 2024-03-26T01:57:28.037 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:27 smithi055 ceph-mon[30619]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:28.037 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:27 smithi055 ceph-mon[30619]: osd.0 [v2:172.21.15.3:6802/388847396,v1:172.21.15.3:6803/388847396] boot 2024-03-26T01:57:28.038 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:27 smithi055 ceph-mon[30619]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T01:57:28.038 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:27 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:28.128 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:27 smithi066 ceph-mon[30259]: purged_snaps scrub starts 2024-03-26T01:57:28.129 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:27 smithi066 ceph-mon[30259]: purged_snaps scrub ok 2024-03-26T01:57:28.129 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:27 smithi066 ceph-mon[30259]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:28.129 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:27 smithi066 ceph-mon[30259]: osd.0 [v2:172.21.15.3:6802/388847396,v1:172.21.15.3:6803/388847396] boot 2024-03-26T01:57:28.129 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:27 smithi066 ceph-mon[30259]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T01:57:28.129 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:27 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:28.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:27 smithi003 ceph-mon[25810]: purged_snaps scrub starts 2024-03-26T01:57:28.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:27 smithi003 ceph-mon[25810]: purged_snaps scrub ok 2024-03-26T01:57:28.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:27 smithi003 ceph-mon[25810]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:28.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:27 smithi003 ceph-mon[25810]: osd.0 [v2:172.21.15.3:6802/388847396,v1:172.21.15.3:6803/388847396] boot 2024-03-26T01:57:28.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:27 smithi003 ceph-mon[25810]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T01:57:28.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:27 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:28.579 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi055:/dev/nvme4n1 2024-03-26T01:57:28.817 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T01:57:28.938 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:28 smithi055 ceph-mon[30619]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T01:57:29.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:28 smithi003 ceph-mon[25810]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T01:57:29.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:28 smithi066 ceph-mon[30259]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T01:57:30.066 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:29 smithi055 ceph-mon[30619]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:30.066 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:29 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:30.066 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:29 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:30.066 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:29 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:57:30.066 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:29 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:30.067 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:29 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:30.067 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:29 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:30.067 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:29 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:30.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:29 smithi003 ceph-mon[25810]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:30.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:29 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:30.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:29 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:30.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:29 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:57:30.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:29 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:30.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:29 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:30.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:29 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:30.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:29 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:30.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:29 smithi066 ceph-mon[30259]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:30.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:29 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:30.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:29 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:30.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:29 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:57:30.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:29 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:30.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:29 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:30.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:29 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:30.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:29 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:31.129 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:30 smithi055 ceph-mon[30619]: Detected new or changed devices on smithi003 2024-03-26T01:57:31.129 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:30 smithi055 ceph-mon[30619]: Adjusting osd_memory_target on smithi003 to 17093M 2024-03-26T01:57:31.129 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:30 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:57:31.129 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:30 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:57:31.129 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:30 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:31.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:30 smithi003 ceph-mon[25810]: Detected new or changed devices on smithi003 2024-03-26T01:57:31.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:30 smithi003 ceph-mon[25810]: Adjusting osd_memory_target on smithi003 to 17093M 2024-03-26T01:57:31.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:30 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:57:31.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:30 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:57:31.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:30 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:31.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:30 smithi066 ceph-mon[30259]: Detected new or changed devices on smithi003 2024-03-26T01:57:31.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:30 smithi066 ceph-mon[30259]: Adjusting osd_memory_target on smithi003 to 17093M 2024-03-26T01:57:31.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:30 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:57:31.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:30 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:57:31.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:30 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:32.019 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:31 smithi055 ceph-mon[30619]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:32.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:31 smithi055 ceph-mon[30619]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:32.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:31 smithi003 ceph-mon[25810]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:32.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:31 smithi003 ceph-mon[25810]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:32.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:31 smithi066 ceph-mon[30259]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:32.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:31 smithi066 ceph-mon[30259]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:34.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:33 smithi003 ceph-mon[25810]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:34.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:33 smithi003 ceph-mon[25810]: from='client.? 172.21.15.55:0/517646114' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ce006b9-8026-4e11-b436-a97e9453b8a4"}]: dispatch 2024-03-26T01:57:34.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:33 smithi003 ceph-mon[25810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ce006b9-8026-4e11-b436-a97e9453b8a4"}]: dispatch 2024-03-26T01:57:34.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:33 smithi003 ceph-mon[25810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5ce006b9-8026-4e11-b436-a97e9453b8a4"}]': finished 2024-03-26T01:57:34.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:33 smithi003 ceph-mon[25810]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T01:57:34.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:33 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:57:34.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:33 smithi066 ceph-mon[30259]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:34.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:33 smithi066 ceph-mon[30259]: from='client.? 172.21.15.55:0/517646114' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ce006b9-8026-4e11-b436-a97e9453b8a4"}]: dispatch 2024-03-26T01:57:34.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:33 smithi066 ceph-mon[30259]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ce006b9-8026-4e11-b436-a97e9453b8a4"}]: dispatch 2024-03-26T01:57:34.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:33 smithi066 ceph-mon[30259]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5ce006b9-8026-4e11-b436-a97e9453b8a4"}]': finished 2024-03-26T01:57:34.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:33 smithi066 ceph-mon[30259]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T01:57:34.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:33 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:57:34.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:33 smithi055 ceph-mon[30619]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:34.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:33 smithi055 ceph-mon[30619]: from='client.? 172.21.15.55:0/517646114' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ce006b9-8026-4e11-b436-a97e9453b8a4"}]: dispatch 2024-03-26T01:57:34.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:33 smithi055 ceph-mon[30619]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ce006b9-8026-4e11-b436-a97e9453b8a4"}]: dispatch 2024-03-26T01:57:34.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:33 smithi055 ceph-mon[30619]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5ce006b9-8026-4e11-b436-a97e9453b8a4"}]': finished 2024-03-26T01:57:34.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:33 smithi055 ceph-mon[30619]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T01:57:34.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:33 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:57:35.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:34 smithi003 ceph-mon[25810]: from='client.? 172.21.15.55:0/1136459033' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:57:35.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:34 smithi066 ceph-mon[30259]: from='client.? 172.21.15.55:0/1136459033' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:57:35.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:34 smithi055 ceph-mon[30619]: from='client.? 172.21.15.55:0/1136459033' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:57:36.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:35 smithi003 ceph-mon[25810]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:36.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:35 smithi066 ceph-mon[30259]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:36.271 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:35 smithi055 ceph-mon[30619]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:38.130 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:37 smithi055 ceph-mon[30619]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:38.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:37 smithi003 ceph-mon[25810]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:38.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:37 smithi066 ceph-mon[30259]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:40.131 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:39 smithi055 ceph-mon[30619]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:40.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:39 smithi003 ceph-mon[25810]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:40.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:39 smithi066 ceph-mon[30259]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:41.108 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:40 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T01:57:41.109 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:40 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:41.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:40 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T01:57:41.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:40 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:41.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:40 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T01:57:41.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:40 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:42.097 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:41 smithi055 ceph-mon[30619]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:42.098 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:41 smithi055 ceph-mon[30619]: Deploying daemon osd.1 on smithi055 2024-03-26T01:57:42.131 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:41 smithi003 ceph-mon[25810]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:42.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:41 smithi003 ceph-mon[25810]: Deploying daemon osd.1 on smithi055 2024-03-26T01:57:42.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:41 smithi066 ceph-mon[30259]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:42.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:41 smithi066 ceph-mon[30259]: Deploying daemon osd.1 on smithi055 2024-03-26T01:57:44.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:43 smithi003 ceph-mon[25810]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:44.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:43 smithi066 ceph-mon[30259]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:44.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:43 smithi055 ceph-mon[30619]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:46.096 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:45 smithi055 ceph-mon[30619]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:46.096 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:45 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:46.097 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:45 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:46.097 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:45 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:46.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:45 smithi003 ceph-mon[25810]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:46.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:45 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:46.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:45 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:46.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:45 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:46.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:45 smithi066 ceph-mon[30259]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:46.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:45 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:46.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:45 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:46.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:45 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:47.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:47 smithi055 ceph-mon[30619]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:48.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:47 smithi003 ceph-mon[25810]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:48.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:47 smithi066 ceph-mon[30259]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:49.097 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:48 smithi055 ceph-mon[30619]: from='osd.1 [v2:172.21.15.55:6800/3045481157,v1:172.21.15.55:6801/3045481157]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T01:57:49.097 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:48 smithi055 ceph-mon[30619]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T01:57:49.098 INFO:teuthology.orchestra.run.smithi055.stdout:Created osd(s) 1 on host 'smithi055' 2024-03-26T01:57:49.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:48 smithi003 ceph-mon[25810]: from='osd.1 [v2:172.21.15.55:6800/3045481157,v1:172.21.15.55:6801/3045481157]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T01:57:49.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:48 smithi003 ceph-mon[25810]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T01:57:49.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:48 smithi066 ceph-mon[30259]: from='osd.1 [v2:172.21.15.55:6800/3045481157,v1:172.21.15.55:6801/3045481157]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T01:57:49.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:48 smithi066 ceph-mon[30259]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T01:57:49.689 DEBUG:teuthology.orchestra.run.smithi055:osd.1> sudo journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@osd.1.service 2024-03-26T01:57:49.691 INFO:tasks.cephadm:Deploying osd.2 on smithi066 with /dev/nvme4n1... 2024-03-26T01:57:49.691 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-26T01:57:49.953 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T01:57:49.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:49 smithi066 ceph-mon[30259]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:49.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:49 smithi066 ceph-mon[30259]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T01:57:49.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:49 smithi066 ceph-mon[30259]: from='osd.1 [v2:172.21.15.55:6800/3045481157,v1:172.21.15.55:6801/3045481157]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-03-26T01:57:49.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:49 smithi066 ceph-mon[30259]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T01:57:49.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:49 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:57:49.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:49 smithi066 ceph-mon[30259]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-03-26T01:57:49.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:49 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:49.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:49 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:49.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:49 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:49.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:49 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:49.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:49 smithi055 ceph-mon[30619]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:49.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:49 smithi055 ceph-mon[30619]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T01:57:49.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:49 smithi055 ceph-mon[30619]: from='osd.1 [v2:172.21.15.55:6800/3045481157,v1:172.21.15.55:6801/3045481157]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-03-26T01:57:49.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:49 smithi055 ceph-mon[30619]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T01:57:49.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:49 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:57:49.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:49 smithi055 ceph-mon[30619]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-03-26T01:57:49.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:49 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:49.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:49 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:49.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:49 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:49.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:49 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:50.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:49 smithi003 ceph-mon[25810]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:50.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:49 smithi003 ceph-mon[25810]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T01:57:50.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:49 smithi003 ceph-mon[25810]: from='osd.1 [v2:172.21.15.55:6800/3045481157,v1:172.21.15.55:6801/3045481157]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-03-26T01:57:50.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:49 smithi003 ceph-mon[25810]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T01:57:50.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:49 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:57:50.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:49 smithi003 ceph-mon[25810]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-03-26T01:57:50.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:49 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:50.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:49 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:50.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:49 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:50.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:49 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:50.769 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 01:57:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-1[35177]: 2024-03-26T01:57:50.358+0000 7f20b689b700 -1 osd.1 0 waiting for initial osdmap 2024-03-26T01:57:50.770 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 01:57:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-1[35177]: 2024-03-26T01:57:50.361+0000 7f20ade85700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T01:57:51.126 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:50 smithi055 ceph-mon[30619]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-03-26T01:57:51.127 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:50 smithi055 ceph-mon[30619]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T01:57:51.127 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:50 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:57:51.127 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:50 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:57:51.127 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:50 smithi055 ceph-mon[30619]: from='osd.1 ' entity='osd.1' 2024-03-26T01:57:51.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:50 smithi003 ceph-mon[25810]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-03-26T01:57:51.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:50 smithi003 ceph-mon[25810]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T01:57:51.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:50 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:57:51.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:50 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:57:51.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:50 smithi003 ceph-mon[25810]: from='osd.1 ' entity='osd.1' 2024-03-26T01:57:51.221 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:50 smithi066 ceph-mon[30259]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-03-26T01:57:51.221 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:50 smithi066 ceph-mon[30259]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T01:57:51.221 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:50 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:57:51.221 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:50 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:57:51.221 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:50 smithi066 ceph-mon[30259]: from='osd.1 ' entity='osd.1' 2024-03-26T01:57:51.967 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping: /dev/nvme4n1 2024-03-26T01:57:51.967 INFO:teuthology.orchestra.run.smithi066.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-26T01:57:51.967 INFO:teuthology.orchestra.run.smithi066.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-26T01:57:51.967 INFO:teuthology.orchestra.run.smithi066.stderr: stderr: 10+0 records in 2024-03-26T01:57:51.967 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records out 2024-03-26T01:57:51.967 INFO:teuthology.orchestra.run.smithi066.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0197585 s, 531 MB/s 2024-03-26T01:57:51.967 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping successful for: 2024-03-26T01:57:52.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:51 smithi003 ceph-mon[25810]: purged_snaps scrub starts 2024-03-26T01:57:52.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:51 smithi003 ceph-mon[25810]: purged_snaps scrub ok 2024-03-26T01:57:52.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:51 smithi003 ceph-mon[25810]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:52.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:57:52.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:51 smithi003 ceph-mon[25810]: osd.1 [v2:172.21.15.55:6800/3045481157,v1:172.21.15.55:6801/3045481157] boot 2024-03-26T01:57:52.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:51 smithi003 ceph-mon[25810]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T01:57:52.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:57:52.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:51 smithi066 ceph-mon[30259]: purged_snaps scrub starts 2024-03-26T01:57:52.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:51 smithi066 ceph-mon[30259]: purged_snaps scrub ok 2024-03-26T01:57:52.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:51 smithi066 ceph-mon[30259]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:52.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:51 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:57:52.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:51 smithi066 ceph-mon[30259]: osd.1 [v2:172.21.15.55:6800/3045481157,v1:172.21.15.55:6801/3045481157] boot 2024-03-26T01:57:52.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:51 smithi066 ceph-mon[30259]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T01:57:52.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:51 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:57:52.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:51 smithi055 ceph-mon[30619]: purged_snaps scrub starts 2024-03-26T01:57:52.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:51 smithi055 ceph-mon[30619]: purged_snaps scrub ok 2024-03-26T01:57:52.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:51 smithi055 ceph-mon[30619]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:52.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:51 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:57:52.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:51 smithi055 ceph-mon[30619]: osd.1 [v2:172.21.15.55:6800/3045481157,v1:172.21.15.55:6801/3045481157] boot 2024-03-26T01:57:52.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:51 smithi055 ceph-mon[30619]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T01:57:52.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:51 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:57:52.500 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi066:/dev/nvme4n1 2024-03-26T01:57:52.744 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T01:57:53.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:53 smithi003 ceph-mon[25810]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:57:53.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:53 smithi003 ceph-mon[25810]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T01:57:53.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:53 smithi003 ceph-mon[25810]: Detected new or changed devices on smithi055 2024-03-26T01:57:53.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:53 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:53.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:53 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:53.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:53 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:57:53.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:53 smithi003 ceph-mon[25810]: Adjusting osd_memory_target on smithi055 to 17093M 2024-03-26T01:57:53.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:53 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:53.642 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:53 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:53.642 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:53 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:53.642 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:53 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:53.642 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:53 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:53.642 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:53 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:53.642 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:53 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:53.642 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:53 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:53.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:53 smithi066 ceph-mon[30259]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:57:53.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:53 smithi066 ceph-mon[30259]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T01:57:53.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:53 smithi066 ceph-mon[30259]: Detected new or changed devices on smithi055 2024-03-26T01:57:53.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:53 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:53.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:53 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:53.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:53 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:57:53.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:53 smithi066 ceph-mon[30259]: Adjusting osd_memory_target on smithi055 to 17093M 2024-03-26T01:57:53.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:53 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:53.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:53 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:53.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:53 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:53.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:53 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:53.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:53 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:53.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:53 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:53.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:53 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:53.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:53 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:53.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:53 smithi055 ceph-mon[30619]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:57:53.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:53 smithi055 ceph-mon[30619]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T01:57:53.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:53 smithi055 ceph-mon[30619]: Detected new or changed devices on smithi055 2024-03-26T01:57:53.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:53 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:53.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:53 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:53.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:53 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:57:53.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:53 smithi055 ceph-mon[30619]: Adjusting osd_memory_target on smithi055 to 17093M 2024-03-26T01:57:53.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:53 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:53.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:53 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:53.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:53 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:53.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:53 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:53.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:53 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:53.771 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:53 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:53.771 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:53 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:53.771 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:53 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:57:55.442 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:55 smithi066 ceph-mon[30259]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:57:55.442 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:55 smithi066 ceph-mon[30259]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:55.442 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:55 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:57:55.442 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:55 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:57:55.442 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:55 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:55.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:55 smithi003 ceph-mon[25810]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:57:55.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:55 smithi003 ceph-mon[25810]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:55.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:55 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:57:55.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:55 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:57:55.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:55 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:55.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:55 smithi055 ceph-mon[30619]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:57:55.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:55 smithi055 ceph-mon[30619]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:55.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:55 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:57:55.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:55 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:57:55.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:55 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:57.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:57 smithi003 ceph-mon[25810]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:57:57.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:57 smithi003 ceph-mon[25810]: from='client.? 172.21.15.66:0/3197485724' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d9d6781-7a40-43f3-9068-bd0e4038b318"}]: dispatch 2024-03-26T01:57:57.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:57 smithi003 ceph-mon[25810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d9d6781-7a40-43f3-9068-bd0e4038b318"}]: dispatch 2024-03-26T01:57:57.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:57 smithi003 ceph-mon[25810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2d9d6781-7a40-43f3-9068-bd0e4038b318"}]': finished 2024-03-26T01:57:57.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:57 smithi003 ceph-mon[25810]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T01:57:57.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:57 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:57:57.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:57 smithi066 ceph-mon[30259]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:57:57.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:57 smithi066 ceph-mon[30259]: from='client.? 172.21.15.66:0/3197485724' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d9d6781-7a40-43f3-9068-bd0e4038b318"}]: dispatch 2024-03-26T01:57:57.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:57 smithi066 ceph-mon[30259]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d9d6781-7a40-43f3-9068-bd0e4038b318"}]: dispatch 2024-03-26T01:57:57.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:57 smithi066 ceph-mon[30259]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2d9d6781-7a40-43f3-9068-bd0e4038b318"}]': finished 2024-03-26T01:57:57.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:57 smithi066 ceph-mon[30259]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T01:57:57.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:57 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:57:57.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:57 smithi055 ceph-mon[30619]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:57:57.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:57 smithi055 ceph-mon[30619]: from='client.? 172.21.15.66:0/3197485724' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d9d6781-7a40-43f3-9068-bd0e4038b318"}]: dispatch 2024-03-26T01:57:57.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:57 smithi055 ceph-mon[30619]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d9d6781-7a40-43f3-9068-bd0e4038b318"}]: dispatch 2024-03-26T01:57:57.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:57 smithi055 ceph-mon[30619]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2d9d6781-7a40-43f3-9068-bd0e4038b318"}]': finished 2024-03-26T01:57:57.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:57 smithi055 ceph-mon[30619]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T01:57:57.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:57 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:57:59.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:59 smithi003 ceph-mon[25810]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:57:59.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:57:59 smithi003 ceph-mon[25810]: from='client.? 172.21.15.66:0/1049869302' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:57:59.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:59 smithi066 ceph-mon[30259]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:57:59.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:57:59 smithi066 ceph-mon[30259]: from='client.? 172.21.15.66:0/1049869302' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:57:59.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:59 smithi055 ceph-mon[30619]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:57:59.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:57:59 smithi055 ceph-mon[30619]: from='client.? 172.21.15.66:0/1049869302' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:58:01.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:01 smithi003 ceph-mon[25810]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:01.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:01 smithi066 ceph-mon[30259]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:01.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:01 smithi055 ceph-mon[30619]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:03.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:03 smithi003 ceph-mon[25810]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:03.681 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:03 smithi066 ceph-mon[30259]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:03.767 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:03 smithi055 ceph-mon[30619]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:04.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:04 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T01:58:04.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:04 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:04.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:04 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T01:58:04.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:04 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:04.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:04 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T01:58:04.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:04 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:05.600 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:05 smithi066 ceph-mon[30259]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:05.600 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:05 smithi066 ceph-mon[30259]: Deploying daemon osd.2 on smithi066 2024-03-26T01:58:05.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:05 smithi055 ceph-mon[30619]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:05.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:05 smithi055 ceph-mon[30619]: Deploying daemon osd.2 on smithi066 2024-03-26T01:58:05.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:05 smithi003 ceph-mon[25810]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:05.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:05 smithi003 ceph-mon[25810]: Deploying daemon osd.2 on smithi066 2024-03-26T01:58:07.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:07 smithi066 ceph-mon[30259]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:07.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:07 smithi055 ceph-mon[30619]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:07.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:07 smithi003 ceph-mon[25810]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:09.670 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:09 smithi066 ceph-mon[30259]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:09.693 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:09 smithi003 ceph-mon[25810]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:09.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:09 smithi055 ceph-mon[30619]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:11.219 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:10 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:11.219 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:10 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:11.219 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:10 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:11.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:10 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:11.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:10 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:11.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:10 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:11.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:10 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:11.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:10 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:11.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:10 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:12.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:11 smithi066 ceph-mon[30259]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:12.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:11 smithi055 ceph-mon[30619]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:12.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:11 smithi003 ceph-mon[25810]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:13.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:13 smithi066 ceph-mon[30259]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:13.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:13 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:13.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:13 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:13.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:13 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:13.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:13 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:13.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:13 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:13.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:13 smithi003 ceph-mon[25810]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:13.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:13 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:13.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:13 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:13.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:13 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:13.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:13 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:13.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:13 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:13.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:13 smithi055 ceph-mon[30619]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:13.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:13 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:13.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:13 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:13.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:13 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:13.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:13 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:13.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:13 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:13.934 INFO:teuthology.orchestra.run.smithi066.stdout:Created osd(s) 2 on host 'smithi066' 2024-03-26T01:58:14.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:14 smithi066 ceph-mon[30259]: from='osd.2 [v2:172.21.15.66:6800/1716775083,v1:172.21.15.66:6801/1716775083]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T01:58:14.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:14 smithi066 ceph-mon[30259]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T01:58:14.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:14 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:14.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:14 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:14.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:14 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:14.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:14 smithi055 ceph-mon[30619]: from='osd.2 [v2:172.21.15.66:6800/1716775083,v1:172.21.15.66:6801/1716775083]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T01:58:14.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:14 smithi055 ceph-mon[30619]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T01:58:14.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:14 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:14.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:14 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:14.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:14 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:14.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:14 smithi003 ceph-mon[25810]: from='osd.2 [v2:172.21.15.66:6800/1716775083,v1:172.21.15.66:6801/1716775083]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T01:58:14.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:14 smithi003 ceph-mon[25810]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T01:58:14.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:14 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:14.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:14 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:14.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:14 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:14.594 DEBUG:teuthology.orchestra.run.smithi066:osd.2> sudo journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@osd.2.service 2024-03-26T01:58:14.596 INFO:tasks.cephadm:Waiting for 3 OSDs to come up... 2024-03-26T01:58:14.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-26T01:58:14.847 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:58:15.138 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:15 smithi066 ceph-mon[30259]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:15.138 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:15 smithi066 ceph-mon[30259]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T01:58:15.138 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:15 smithi066 ceph-mon[30259]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T01:58:15.138 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:15 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:15.138 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:15 smithi066 ceph-mon[30259]: from='osd.2 [v2:172.21.15.66:6800/1716775083,v1:172.21.15.66:6801/1716775083]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T01:58:15.138 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:15 smithi066 ceph-mon[30259]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T01:58:15.372 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:15 smithi003 ceph-mon[25810]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:15.373 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:15 smithi003 ceph-mon[25810]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T01:58:15.373 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:15 smithi003 ceph-mon[25810]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T01:58:15.373 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:15 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:15.373 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:15 smithi003 ceph-mon[25810]: from='osd.2 [v2:172.21.15.66:6800/1716775083,v1:172.21.15.66:6801/1716775083]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T01:58:15.373 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:15 smithi003 ceph-mon[25810]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T01:58:15.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:15 smithi055 ceph-mon[30619]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:15.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:15 smithi055 ceph-mon[30619]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T01:58:15.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:15 smithi055 ceph-mon[30619]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T01:58:15.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:15 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:15.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:15 smithi055 ceph-mon[30619]: from='osd.2 [v2:172.21.15.66:6800/1716775083,v1:172.21.15.66:6801/1716775083]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T01:58:15.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:15 smithi055 ceph-mon[30619]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T01:58:15.782 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 01:58:15 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-2[35524]: 2024-03-26T01:58:15.533+0000 7f8072dd2700 -1 osd.2 0 waiting for initial osdmap 2024-03-26T01:58:15.782 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 01:58:15 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-2[35524]: 2024-03-26T01:58:15.536+0000 7f806a3bc700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T01:58:16.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:16 smithi003 ceph-mon[25810]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-26T01:58:16.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:16 smithi003 ceph-mon[25810]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T01:58:16.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:16 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:16.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:16 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:16.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:16 smithi003 ceph-mon[25810]: from='osd.2 ' entity='osd.2' 2024-03-26T01:58:16.318 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:16 smithi066 ceph-mon[30259]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-26T01:58:16.318 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:16 smithi066 ceph-mon[30259]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T01:58:16.318 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:16 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:16.318 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:16 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:16.319 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:16 smithi066 ceph-mon[30259]: from='osd.2 ' entity='osd.2' 2024-03-26T01:58:16.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:16 smithi055 ceph-mon[30619]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-26T01:58:16.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:16 smithi055 ceph-mon[30619]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T01:58:16.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:16 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:16.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:16 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:16.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:16 smithi055 ceph-mon[30619]: from='osd.2 ' entity='osd.2' 2024-03-26T01:58:16.846 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:58:17.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:17 smithi003 ceph-mon[25810]: purged_snaps scrub starts 2024-03-26T01:58:17.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:17 smithi003 ceph-mon[25810]: purged_snaps scrub ok 2024-03-26T01:58:17.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:17 smithi003 ceph-mon[25810]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:17.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:17 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:17.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:17 smithi003 ceph-mon[25810]: osd.2 [v2:172.21.15.66:6800/1716775083,v1:172.21.15.66:6801/1716775083] boot 2024-03-26T01:58:17.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:17 smithi003 ceph-mon[25810]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T01:58:17.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:17 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:17.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:17 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/4221570159' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T01:58:17.316 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:17 smithi066 ceph-mon[30259]: purged_snaps scrub starts 2024-03-26T01:58:17.316 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:17 smithi066 ceph-mon[30259]: purged_snaps scrub ok 2024-03-26T01:58:17.316 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:17 smithi066 ceph-mon[30259]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:17.317 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:17 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:17.317 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:17 smithi066 ceph-mon[30259]: osd.2 [v2:172.21.15.66:6800/1716775083,v1:172.21.15.66:6801/1716775083] boot 2024-03-26T01:58:17.317 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:17 smithi066 ceph-mon[30259]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T01:58:17.317 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:17 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:17.317 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:17 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/4221570159' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T01:58:17.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:17 smithi055 ceph-mon[30619]: purged_snaps scrub starts 2024-03-26T01:58:17.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:17 smithi055 ceph-mon[30619]: purged_snaps scrub ok 2024-03-26T01:58:17.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:17 smithi055 ceph-mon[30619]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:17.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:17 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:17.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:17 smithi055 ceph-mon[30619]: osd.2 [v2:172.21.15.66:6800/1716775083,v1:172.21.15.66:6801/1716775083] boot 2024-03-26T01:58:17.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:17 smithi055 ceph-mon[30619]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T01:58:17.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:17 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:17.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:17 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/4221570159' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T01:58:17.624 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":18,"num_osds":3,"num_up_osds":3,"osd_up_since":1711418296,"num_in_osds":3,"osd_in_since":1711418277,"num_remapped_pgs":0} 2024-03-26T01:58:17.625 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T01:58:17.870 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:58:19.080 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:19 smithi003 ceph-mon[25810]: pgmap v83: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:19.080 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:19 smithi003 ceph-mon[25810]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T01:58:19.080 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:19 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T01:58:19.080 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:19 smithi003 ceph-mon[25810]: Detected new or changed devices on smithi066 2024-03-26T01:58:19.080 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:19 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:19.080 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:19 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:19.080 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:19 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:58:19.080 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:19 smithi003 ceph-mon[25810]: Adjusting osd_memory_target on smithi066 to 17088M 2024-03-26T01:58:19.081 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:19 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:19.081 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:19 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:19.081 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:19 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:19.081 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:19 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:19.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:19 smithi066 ceph-mon[30259]: pgmap v83: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:19.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:19 smithi066 ceph-mon[30259]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T01:58:19.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:19 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T01:58:19.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:19 smithi066 ceph-mon[30259]: Detected new or changed devices on smithi066 2024-03-26T01:58:19.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:19 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:19.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:19 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:19.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:19 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:58:19.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:19 smithi066 ceph-mon[30259]: Adjusting osd_memory_target on smithi066 to 17088M 2024-03-26T01:58:19.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:19 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:19.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:19 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:19.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:19 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:19.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:19 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:19.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:19 smithi055 ceph-mon[30619]: pgmap v83: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:19.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:19 smithi055 ceph-mon[30619]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T01:58:19.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:19 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T01:58:19.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:19 smithi055 ceph-mon[30619]: Detected new or changed devices on smithi066 2024-03-26T01:58:19.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:19 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:19.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:19 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:19.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:19 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:58:19.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:19 smithi055 ceph-mon[30619]: Adjusting osd_memory_target on smithi066 to 17088M 2024-03-26T01:58:19.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:19 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:19.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:19 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:19.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:19 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:19.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:19 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:19.971 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:58:19.971 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"fsid":"b95eae18-eb13-11ee-95d0-87774f69a715","created":"2024-03-26T01:54:47.555614+0000","modified":"2024-03-26T01:58:19.055798+0000","last_up_change":"2024-03-26T01:58:16.536620+0000","last_in_change":"2024-03-26T01:57:57.288513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T01:58:18.340237+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b217e513-44cb-41f3-a109-5a3688f2c8ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":388847396},{"type":"v1","addr":"172.21.15.3:6803","nonce":388847396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":388847396},{"type":"v1","addr":"172.21.15.3:6805","nonce":388847396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":388847396},{"type":"v1","addr":"172.21.15.3:6809","nonce":388847396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":388847396},{"type":"v1","addr":"172.21.15.3:6807","nonce":388847396}]},"public_addr":"172.21.15.3:6803/388847396","cluster_addr":"172.21.15.3:6805/388847396","heartbeat_back_addr":"172.21.15.3:6809/388847396","heartbeat_front_addr":"172.21.15.3:6807/388847396","state":["exists","up"]},{"osd":1,"uuid":"5ce006b9-8026-4e11-b436-a97e9453b8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":3045481157},{"type":"v1","addr":"172.21.15.55:6801","nonce":3045481157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":3045481157},{"type":"v1","addr":"172.21.15.55:6803","nonce":3045481157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":3045481157},{"type":"v1","addr":"172.21.15.55:6807","nonce":3045481157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":3045481157},{"type":"v1","addr":"172.21.15.55:6805","nonce":3045481157}]},"public_addr":"172.21.15.55:6801/3045481157","cluster_addr":"172.21.15.55:6803/3045481157","heartbeat_back_addr":"172.21.15.55:6807/3045481157","heartbeat_front_addr":"172.21.15.55:6805/3045481157","state":["exists","up"]},{"osd":2,"uuid":"2d9d6781-7a40-43f3-9068-bd0e4038b318","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":1716775083},{"type":"v1","addr":"172.21.15.66:6801","nonce":1716775083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":1716775083},{"type":"v1","addr":"172.21.15.66:6803","nonce":1716775083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":1716775083},{"type":"v1","addr":"172.21.15.66:6807","nonce":1716775083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":1716775083},{"type":"v1","addr":"172.21.15.66:6805","nonce":1716775083}]},"public_addr":"172.21.15.66:6801/1716775083","cluster_addr":"172.21.15.66:6803/1716775083","heartbeat_back_addr":"172.21.15.66:6807/1716775083","heartbeat_front_addr":"172.21.15.66:6805/1716775083","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:57:25.422890+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:57:49.555207+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:58:14.030714+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/972878027":"2024-03-27T01:55:47.993042+0000","172.21.15.3:6800/372539062":"2024-03-27T01:55:47.993042+0000","172.21.15.3:0/3566218324":"2024-03-27T01:55:20.887332+0000","172.21.15.3:6800/1304425570":"2024-03-27T01:55:20.887332+0000","172.21.15.3:6801/372539062":"2024-03-27T01:55:47.993042+0000","172.21.15.3:6801/1304425570":"2024-03-27T01:55:20.887332+0000","172.21.15.3:0/2863586814":"2024-03-27T01:55:47.993042+0000","172.21.15.3:0/2403034937":"2024-03-27T01:55:20.887332+0000","172.21.15.3:0/949463668":"2024-03-27T01:55:20.887332+0000","172.21.15.3:0/2144919668":"2024-03-27T01:55:47.993042+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T01:58:20.327 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:20 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T01:58:20.328 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:20 smithi003 ceph-mon[25810]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T01:58:20.328 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:20 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T01:58:20.328 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:20 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/568697811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T01:58:20.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:20 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T01:58:20.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:20 smithi066 ceph-mon[30259]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T01:58:20.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:20 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T01:58:20.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:20 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/568697811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T01:58:20.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:20 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T01:58:20.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:20 smithi055 ceph-mon[30619]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T01:58:20.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:20 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T01:58:20.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:20 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/568697811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T01:58:20.612 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:58:20 smithi003 sudo[46349]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-26T01:58:20.612 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:58:20 smithi003 sudo[46349]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:20.612 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:58:20 smithi003 sudo[46349]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:20.612 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:58:20 smithi003 sudo[46349]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:20.641 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T01:58:18.340237+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T01:58:20.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-26T01:58:20.880 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:58:20.890 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:58:20 smithi003 sudo[46372]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-26T01:58:20.890 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:58:20 smithi003 sudo[46372]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:20.891 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:58:20 smithi003 sudo[46372]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:20.891 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:58:20 smithi003 sudo[46372]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:21.241 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 01:58:20 smithi055 sudo[40837]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-26T01:58:21.241 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 01:58:20 smithi055 sudo[40837]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:21.241 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 01:58:20 smithi055 sudo[40837]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:21.241 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 01:58:20 smithi055 sudo[40837]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:21.242 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:21 smithi055 ceph-mon[30619]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:21.242 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:21 smithi055 ceph-mon[30619]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T01:58:21.242 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:21 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T01:58:21.242 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:21 smithi055 ceph-mon[30619]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T01:58:21.242 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:21 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:21.264 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:21 smithi003 ceph-mon[25810]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:21.264 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:21 smithi003 ceph-mon[25810]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T01:58:21.264 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:21 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T01:58:21.264 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:21 smithi003 ceph-mon[25810]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T01:58:21.264 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:21 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:21.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:21 smithi066 ceph-mon[30259]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:21.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:21 smithi066 ceph-mon[30259]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T01:58:21.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:21 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T01:58:21.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:21 smithi066 ceph-mon[30259]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T01:58:21.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:21 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:21.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 01:58:21 smithi055 sudo[40840]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-26T01:58:21.520 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 01:58:21 smithi055 sudo[40840]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:21.520 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 01:58:21 smithi055 sudo[40840]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:21.520 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 01:58:21 smithi055 sudo[40840]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:21.977 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 01:58:21 smithi066 sudo[41119]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-26T01:58:21.978 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 01:58:21 smithi066 sudo[41119]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:21.978 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 01:58:21 smithi066 sudo[41119]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:21.978 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 01:58:21 smithi066 sudo[41119]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:22.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:22 smithi003 ceph-mon[25810]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T01:58:22.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:22 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:22.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:22 smithi003 ceph-mon[25810]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-26T01:58:22.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:22 smithi066 ceph-mon[30259]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T01:58:22.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:22 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:22.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:22 smithi066 ceph-mon[30259]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-26T01:58:22.478 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 01:58:22 smithi066 sudo[41122]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-26T01:58:22.478 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 01:58:22 smithi066 sudo[41122]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:22.478 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 01:58:22 smithi066 sudo[41122]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:22.478 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 01:58:22 smithi066 sudo[41122]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:22.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:22 smithi055 ceph-mon[30619]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T01:58:22.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:22 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:22.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:22 smithi055 ceph-mon[30619]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-26T01:58:22.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:22 smithi003 sudo[46523]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T01:58:22.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:22 smithi003 sudo[46523]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:22.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:22 smithi003 sudo[46523]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:22.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:22 smithi003 sudo[46523]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:22.691 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-03-26T01:58:22.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:22 smithi003 sudo[46560]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T01:58:22.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:22 smithi003 sudo[46560]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:22.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:22 smithi003 sudo[46560]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:22.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:22 smithi003 sudo[46560]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:23.280 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi003 2024-03-26T01:58:23.280 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi003=foo.a' 2024-03-26T01:58:23.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 sudo[41125]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T01:58:23.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 sudo[41125]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:23.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 sudo[41125]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:23.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 sudo[41125]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:23.536 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T01:58:23.917 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 sudo[41201]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T01:58:23.917 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 sudo[41201]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:23.917 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 sudo[41201]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:23.917 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 sudo[41201]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:23.918 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 ceph-mon[30259]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:23.918 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:23.918 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 ceph-mon[30259]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:58:23.918 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/2659992793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T01:58:23.918 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 ceph-mon[30259]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:58:23.918 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 ceph-mon[30259]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T01:58:23.918 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 ceph-mon[30259]: Cluster is now healthy 2024-03-26T01:58:23.918 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:58:23.918 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:58:23.918 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:58:23.918 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:23.918 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:58:23.918 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:58:23.919 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:23 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:58:23.948 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:23 smithi055 ceph-mon[30619]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:23.948 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:23 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:23.948 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:23 smithi055 ceph-mon[30619]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:58:23.948 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:23 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/2659992793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T01:58:23.949 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:23 smithi055 ceph-mon[30619]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:58:23.949 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:23 smithi055 ceph-mon[30619]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T01:58:23.949 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:23 smithi055 ceph-mon[30619]: Cluster is now healthy 2024-03-26T01:58:23.949 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:23 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:58:23.949 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:23 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:58:23.949 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:23 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:58:23.949 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:23 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:23.949 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:23 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:58:23.949 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:23 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:58:23.949 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:23 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:58:23.949 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:23 smithi055 sudo[40843]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T01:58:24.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:23 smithi003 ceph-mon[25810]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:24.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:23 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:24.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:23 smithi003 ceph-mon[25810]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:58:24.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:23 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/2659992793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T01:58:24.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:23 smithi003 ceph-mon[25810]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:58:24.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:23 smithi003 ceph-mon[25810]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T01:58:24.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:23 smithi003 ceph-mon[25810]: Cluster is now healthy 2024-03-26T01:58:24.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:23 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:58:24.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:23 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:58:24.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:23 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:58:24.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:23 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:24.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:23 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:58:24.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:23 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:58:24.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:23 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:58:24.213 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:23 smithi055 sudo[40843]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:24.213 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:23 smithi055 sudo[40843]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:24.213 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:23 smithi055 sudo[40843]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:24.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:24 smithi055 sudo[40846]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T01:58:24.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:24 smithi055 sudo[40846]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:24.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:24 smithi055 sudo[40846]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:24.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:24 smithi055 sudo[40846]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:24.940 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:24 smithi066 ceph-mon[30259]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:58:24.940 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:24 smithi066 ceph-mon[30259]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:58:24.941 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:24 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:24.941 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:24 smithi066 ceph-mon[30259]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:58:24.941 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:24 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:58:24.941 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:24 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:58:24.941 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:24 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:58:24.941 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:24 smithi066 ceph-mon[30259]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:58:24.941 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:24 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:25.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:24 smithi055 ceph-mon[30619]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:58:25.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:24 smithi055 ceph-mon[30619]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:58:25.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:24 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:25.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:24 smithi055 ceph-mon[30619]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:58:25.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:24 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:58:25.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:24 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:58:25.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:24 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:58:25.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:24 smithi055 ceph-mon[30619]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:58:25.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:24 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:25.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:24 smithi003 ceph-mon[25810]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:58:25.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:24 smithi003 ceph-mon[25810]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:58:25.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:24 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:25.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:24 smithi003 ceph-mon[25810]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:58:25.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:24 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:58:25.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:24 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:58:25.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:24 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:58:25.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:24 smithi003 ceph-mon[25810]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:58:25.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:24 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:25.400 INFO:teuthology.orchestra.run.smithi066.stdout:Scheduled rgw.foo.a update... 2024-03-26T01:58:25.950 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:25 smithi066 ceph-mon[30259]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:25.950 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:25 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:25.950 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:25 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:25.981 DEBUG:teuthology.orchestra.run.smithi003:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@rgw.foo.a.service 2024-03-26T01:58:25.983 INFO:tasks.cephadm:Adding prometheus.a on smithi055 2024-03-26T01:58:25.984 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph orch apply prometheus '1;smithi055=a' 2024-03-26T01:58:26.009 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:25 smithi003 ceph-mon[25810]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:26.009 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:25 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:26.009 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:25 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:26.019 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:25 smithi055 ceph-mon[30619]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:26.019 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:25 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:26.019 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:25 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:26.236 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T01:58:26.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:26 smithi066 ceph-mon[30259]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi003=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:26.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:26 smithi066 ceph-mon[30259]: Saving service rgw.foo.a spec with placement smithi003=foo.a;count:1 2024-03-26T01:58:26.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:26 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:26.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:26 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:26.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:26 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:26.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:26 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T01:58:26.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:26 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T01:58:26.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:26 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:26.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:26 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:26.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:26 smithi003 ceph-mon[25810]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi003=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:26.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:26 smithi003 ceph-mon[25810]: Saving service rgw.foo.a spec with placement smithi003=foo.a;count:1 2024-03-26T01:58:26.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:26 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:26.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:26 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:26.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:26 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:26.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:26 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T01:58:26.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:26 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T01:58:26.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:26 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:26.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:26 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:27.019 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:26 smithi055 ceph-mon[30619]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi003=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:27.019 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:26 smithi055 ceph-mon[30619]: Saving service rgw.foo.a spec with placement smithi003=foo.a;count:1 2024-03-26T01:58:27.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:26 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:27.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:26 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:27.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:26 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:27.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:26 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T01:58:27.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:26 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T01:58:27.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:26 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:27.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:26 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:27.862 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:27 smithi003 ceph-mon[25810]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:27.862 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:27 smithi003 ceph-mon[25810]: Deploying daemon rgw.foo.a on smithi003 2024-03-26T01:58:27.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:27 smithi066 ceph-mon[30259]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:27.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:27 smithi066 ceph-mon[30259]: Deploying daemon rgw.foo.a on smithi003 2024-03-26T01:58:28.019 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:27 smithi055 ceph-mon[30619]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:28.019 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:27 smithi055 ceph-mon[30619]: Deploying daemon rgw.foo.a on smithi003 2024-03-26T01:58:28.109 INFO:teuthology.orchestra.run.smithi066.stdout:Scheduled prometheus update... 2024-03-26T01:58:28.140 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 26 01:58:28 smithi003 systemd[1]: Starting Ceph rgw.foo.a for b95eae18-eb13-11ee-95d0-87774f69a715... 2024-03-26T01:58:28.593 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 26 01:58:28 smithi003 podman[46882]: 2024-03-26 01:58:28.257775264 +0000 UTC m=+0.110214817 container create 961bfb838631da66ab03242e8bc719af43b123a5f5b4bee22b1ef2bda67c8f7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-rgw-foo-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True) 2024-03-26T01:58:28.594 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 26 01:58:28 smithi003 podman[46882]: 2024-03-26 01:58:28.165202936 +0000 UTC m=+0.017642517 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T01:58:28.594 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 26 01:58:28 smithi003 podman[46882]: 2024-03-26 01:58:28.485635124 +0000 UTC m=+0.338074680 container init 961bfb838631da66ab03242e8bc719af43b123a5f5b4bee22b1ef2bda67c8f7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-rgw-foo-a, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-5e70124) 2024-03-26T01:58:28.594 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 26 01:58:28 smithi003 podman[46882]: 2024-03-26 01:58:28.491337696 +0000 UTC m=+0.343777259 container start 961bfb838631da66ab03242e8bc719af43b123a5f5b4bee22b1ef2bda67c8f7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-rgw-foo-a, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T01:58:28.741 DEBUG:teuthology.orchestra.run.smithi055:prometheus.a> sudo journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@prometheus.a.service 2024-03-26T01:58:28.746 INFO:tasks.cephadm:Adding node-exporter.a on smithi003 2024-03-26T01:58:28.746 INFO:tasks.cephadm:Adding node-exporter.b on smithi055 2024-03-26T01:58:28.746 INFO:tasks.cephadm:Adding node-exporter.c on smithi066 2024-03-26T01:58:28.746 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph orch apply node-exporter '3;smithi003=a;smithi055=b;smithi066=c' 2024-03-26T01:58:28.890 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 26 01:58:28 smithi003 bash[46882]: 961bfb838631da66ab03242e8bc719af43b123a5f5b4bee22b1ef2bda67c8f7a 2024-03-26T01:58:28.891 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 26 01:58:28 smithi003 systemd[1]: Started Ceph rgw.foo.a for b95eae18-eb13-11ee-95d0-87774f69a715. 2024-03-26T01:58:28.993 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T01:58:29.364 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:29 smithi066 ceph-mon[30259]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 392 KiB/s wr, 25 op/s 2024-03-26T01:58:29.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:29 smithi066 ceph-mon[30259]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi055=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:29.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:29 smithi066 ceph-mon[30259]: Saving service prometheus spec with placement smithi055=a;count:1 2024-03-26T01:58:29.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:29 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:29.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:29 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:29.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:29 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:29.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:29 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:29.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:29 smithi066 ceph-mon[30259]: Saving service rgw.foo.a spec with placement smithi003=foo.a;count:1 2024-03-26T01:58:29.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:29 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:29.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:29 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:29.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:29 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:29.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:29 smithi003 ceph-mon[25810]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 392 KiB/s wr, 25 op/s 2024-03-26T01:58:29.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:29 smithi003 ceph-mon[25810]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi055=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:29.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:29 smithi003 ceph-mon[25810]: Saving service prometheus spec with placement smithi055=a;count:1 2024-03-26T01:58:29.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:29 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:29.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:29 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:29.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:29 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:29.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:29 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:29.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:29 smithi003 ceph-mon[25810]: Saving service rgw.foo.a spec with placement smithi003=foo.a;count:1 2024-03-26T01:58:29.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:29 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:29.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:29 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:29.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:29 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:29.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:29 smithi055 ceph-mon[30619]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 392 KiB/s wr, 25 op/s 2024-03-26T01:58:29.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:29 smithi055 ceph-mon[30619]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi055=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:29.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:29 smithi055 ceph-mon[30619]: Saving service prometheus spec with placement smithi055=a;count:1 2024-03-26T01:58:29.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:29 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:29.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:29 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:29.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:29 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:29.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:29 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:29.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:29 smithi055 ceph-mon[30619]: Saving service rgw.foo.a spec with placement smithi003=foo.a;count:1 2024-03-26T01:58:29.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:29 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:29.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:29 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:29.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:29 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:30.389 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:30 smithi066 ceph-mon[30259]: osdmap e23: 3 total, 3 up, 3 in 2024-03-26T01:58:30.390 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:30 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/3885555605' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T01:58:30.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:30 smithi003 ceph-mon[25810]: osdmap e23: 3 total, 3 up, 3 in 2024-03-26T01:58:30.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:30 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/3885555605' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T01:58:30.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:30 smithi055 ceph-mon[30619]: osdmap e23: 3 total, 3 up, 3 in 2024-03-26T01:58:30.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:30 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/3885555605' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T01:58:30.891 INFO:teuthology.orchestra.run.smithi066.stdout:Scheduled node-exporter update... 2024-03-26T01:58:31.216 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:31 smithi066 ceph-mon[30259]: pgmap v94: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 5.6 KiB/s rd, 294 KiB/s wr, 18 op/s 2024-03-26T01:58:31.216 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:31 smithi066 ceph-mon[30259]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T01:58:31.216 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:31 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/3885555605' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T01:58:31.216 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:31 smithi066 ceph-mon[30259]: osdmap e24: 3 total, 3 up, 3 in 2024-03-26T01:58:31.216 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:31 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:31.216 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:31 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:31.216 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:31 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:31.216 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:31 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:31.216 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:31 smithi066 ceph-mon[30259]: Checking dashboard <-> RGW credentials 2024-03-26T01:58:31.217 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:31 smithi066 ceph-mon[30259]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi003=a;smithi055=b;smithi066=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:31.217 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:31 smithi066 ceph-mon[30259]: Saving service node-exporter spec with placement smithi003=a;smithi055=b;smithi066=c;count:3 2024-03-26T01:58:31.217 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:31 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:31.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:31 smithi003 ceph-mon[25810]: pgmap v94: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 5.6 KiB/s rd, 294 KiB/s wr, 18 op/s 2024-03-26T01:58:31.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:31 smithi003 ceph-mon[25810]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T01:58:31.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:31 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/3885555605' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T01:58:31.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:31 smithi003 ceph-mon[25810]: osdmap e24: 3 total, 3 up, 3 in 2024-03-26T01:58:31.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:31 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:31.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:31 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:31.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:31 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:31.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:31 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:31.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:31 smithi003 ceph-mon[25810]: Checking dashboard <-> RGW credentials 2024-03-26T01:58:31.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:31 smithi003 ceph-mon[25810]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi003=a;smithi055=b;smithi066=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:31.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:31 smithi003 ceph-mon[25810]: Saving service node-exporter spec with placement smithi003=a;smithi055=b;smithi066=c;count:3 2024-03-26T01:58:31.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:31 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:31.421 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:31 smithi055 ceph-mon[30619]: pgmap v94: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 5.6 KiB/s rd, 294 KiB/s wr, 18 op/s 2024-03-26T01:58:31.421 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:31 smithi055 ceph-mon[30619]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T01:58:31.421 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:31 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/3885555605' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T01:58:31.421 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:31 smithi055 ceph-mon[30619]: osdmap e24: 3 total, 3 up, 3 in 2024-03-26T01:58:31.421 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:31 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:31.422 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:31 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:31.422 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:31 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:31.422 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:31 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:31.422 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:31 smithi055 ceph-mon[30619]: Checking dashboard <-> RGW credentials 2024-03-26T01:58:31.422 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:31 smithi055 ceph-mon[30619]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi003=a;smithi055=b;smithi066=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:31.422 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:31 smithi055 ceph-mon[30619]: Saving service node-exporter spec with placement smithi003=a;smithi055=b;smithi066=c;count:3 2024-03-26T01:58:31.422 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:31 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:31.492 DEBUG:teuthology.orchestra.run.smithi003:node-exporter.a> sudo journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@node-exporter.a.service 2024-03-26T01:58:31.495 DEBUG:teuthology.orchestra.run.smithi055:node-exporter.b> sudo journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@node-exporter.b.service 2024-03-26T01:58:31.498 DEBUG:teuthology.orchestra.run.smithi066:node-exporter.c> sudo journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@node-exporter.c.service 2024-03-26T01:58:31.500 INFO:tasks.cephadm:Adding alertmanager.a on smithi003 2024-03-26T01:58:31.501 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph orch apply alertmanager '1;smithi003=a' 2024-03-26T01:58:31.792 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T01:58:32.301 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:32 smithi066 ceph-mon[30259]: osdmap e25: 3 total, 3 up, 3 in 2024-03-26T01:58:32.301 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:32 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T01:58:32.301 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:32 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T01:58:32.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:32 smithi055 ceph-mon[30619]: osdmap e25: 3 total, 3 up, 3 in 2024-03-26T01:58:32.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:32 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T01:58:32.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:32 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T01:58:32.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:32 smithi003 ceph-mon[25810]: osdmap e25: 3 total, 3 up, 3 in 2024-03-26T01:58:32.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:32 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T01:58:32.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:32 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T01:58:33.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:33 smithi066 ceph-mon[30259]: pgmap v97: 65 pgs: 11 creating+peering, 43 unknown, 11 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 8.2 KiB/s rd, 392 KiB/s wr, 26 op/s 2024-03-26T01:58:33.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:33 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T01:58:33.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:33 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T01:58:33.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:33 smithi066 ceph-mon[30259]: osdmap e26: 3 total, 3 up, 3 in 2024-03-26T01:58:33.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:33 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:33.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:33 smithi055 ceph-mon[30619]: pgmap v97: 65 pgs: 11 creating+peering, 43 unknown, 11 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 8.2 KiB/s rd, 392 KiB/s wr, 26 op/s 2024-03-26T01:58:33.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:33 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T01:58:33.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:33 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T01:58:33.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:33 smithi055 ceph-mon[30619]: osdmap e26: 3 total, 3 up, 3 in 2024-03-26T01:58:33.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:33 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:33.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:33 smithi003 ceph-mon[25810]: pgmap v97: 65 pgs: 11 creating+peering, 43 unknown, 11 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 8.2 KiB/s rd, 392 KiB/s wr, 26 op/s 2024-03-26T01:58:33.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:33 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T01:58:33.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:33 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T01:58:33.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:33 smithi003 ceph-mon[25810]: osdmap e26: 3 total, 3 up, 3 in 2024-03-26T01:58:33.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:33 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:33.727 INFO:teuthology.orchestra.run.smithi066.stdout:Scheduled alertmanager update... 2024-03-26T01:58:34.369 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:34 smithi066 ceph-mon[30259]: osdmap e27: 3 total, 3 up, 3 in 2024-03-26T01:58:34.369 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:34 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T01:58:34.369 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:34 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T01:58:34.369 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:34 smithi066 ceph-mon[30259]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi003=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:34.369 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:34 smithi066 ceph-mon[30259]: Saving service alertmanager spec with placement smithi003=a;count:1 2024-03-26T01:58:34.369 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:34 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:34.395 DEBUG:teuthology.orchestra.run.smithi003:alertmanager.a> sudo journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@alertmanager.a.service 2024-03-26T01:58:34.397 INFO:tasks.cephadm:Adding grafana.a on smithi055 2024-03-26T01:58:34.398 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph orch apply grafana '1;smithi055=a' 2024-03-26T01:58:34.418 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:34 smithi003 ceph-mon[25810]: osdmap e27: 3 total, 3 up, 3 in 2024-03-26T01:58:34.419 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:34 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T01:58:34.419 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:34 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T01:58:34.419 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:34 smithi003 ceph-mon[25810]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi003=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:34.419 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:34 smithi003 ceph-mon[25810]: Saving service alertmanager spec with placement smithi003=a;count:1 2024-03-26T01:58:34.419 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:34 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:34.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:34 smithi055 ceph-mon[30619]: osdmap e27: 3 total, 3 up, 3 in 2024-03-26T01:58:34.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:34 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T01:58:34.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:34 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T01:58:34.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:34 smithi055 ceph-mon[30619]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi003=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:34.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:34 smithi055 ceph-mon[30619]: Saving service alertmanager spec with placement smithi003=a;count:1 2024-03-26T01:58:34.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:34 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:34.644 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T01:58:35.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:35 smithi066 ceph-mon[30259]: pgmap v100: 97 pgs: 32 unknown, 20 creating+peering, 45 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T01:58:35.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:35 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T01:58:35.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:35 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T01:58:35.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:35 smithi066 ceph-mon[30259]: osdmap e28: 3 total, 3 up, 3 in 2024-03-26T01:58:35.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:35 smithi055 ceph-mon[30619]: pgmap v100: 97 pgs: 32 unknown, 20 creating+peering, 45 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T01:58:35.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:35 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T01:58:35.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:35 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T01:58:35.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:35 smithi055 ceph-mon[30619]: osdmap e28: 3 total, 3 up, 3 in 2024-03-26T01:58:35.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:35 smithi003 ceph-mon[25810]: pgmap v100: 97 pgs: 32 unknown, 20 creating+peering, 45 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T01:58:35.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:35 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T01:58:35.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:35 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T01:58:35.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:35 smithi003 ceph-mon[25810]: osdmap e28: 3 total, 3 up, 3 in 2024-03-26T01:58:36.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:36 smithi066 ceph-mon[30259]: osdmap e29: 3 total, 3 up, 3 in 2024-03-26T01:58:36.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:36 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T01:58:36.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:36 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T01:58:36.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:36 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T01:58:36.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:36 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T01:58:36.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:36 smithi066 ceph-mon[30259]: osdmap e30: 3 total, 3 up, 3 in 2024-03-26T01:58:36.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:36 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T01:58:36.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:36 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T01:58:36.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:36 smithi055 ceph-mon[30619]: osdmap e29: 3 total, 3 up, 3 in 2024-03-26T01:58:36.519 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:36 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T01:58:36.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:36 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T01:58:36.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:36 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T01:58:36.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:36 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T01:58:36.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:36 smithi055 ceph-mon[30619]: osdmap e30: 3 total, 3 up, 3 in 2024-03-26T01:58:36.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:36 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T01:58:36.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:36 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T01:58:36.522 INFO:teuthology.orchestra.run.smithi066.stdout:Scheduled grafana update... 2024-03-26T01:58:36.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:36 smithi003 ceph-mon[25810]: osdmap e29: 3 total, 3 up, 3 in 2024-03-26T01:58:36.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:36 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T01:58:36.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:36 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T01:58:36.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:36 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T01:58:36.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:36 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T01:58:36.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:36 smithi003 ceph-mon[25810]: osdmap e30: 3 total, 3 up, 3 in 2024-03-26T01:58:36.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:36 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T01:58:36.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:36 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T01:58:37.159 DEBUG:teuthology.orchestra.run.smithi055:grafana.a> sudo journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@grafana.a.service 2024-03-26T01:58:37.162 INFO:tasks.cephadm:Setting up client nodes... 2024-03-26T01:58:37.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T01:58:37.491 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:58:37.874 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:37 smithi003 ceph-mon[25810]: pgmap v103: 129 pgs: 64 unknown, 20 creating+peering, 45 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-26T01:58:37.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:37 smithi003 ceph-mon[25810]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi055=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:37.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:37 smithi003 ceph-mon[25810]: Saving service grafana spec with placement smithi055=a;count:1 2024-03-26T01:58:37.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:37 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:37.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:37 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T01:58:37.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:37 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T01:58:37.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:37 smithi003 ceph-mon[25810]: osdmap e31: 3 total, 3 up, 3 in 2024-03-26T01:58:37.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:37 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:37.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:37 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:37.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:37 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:37.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:37 smithi066 ceph-mon[30259]: pgmap v103: 129 pgs: 64 unknown, 20 creating+peering, 45 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-26T01:58:37.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:37 smithi066 ceph-mon[30259]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi055=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:37.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:37 smithi066 ceph-mon[30259]: Saving service grafana spec with placement smithi055=a;count:1 2024-03-26T01:58:37.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:37 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:37.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:37 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T01:58:37.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:37 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T01:58:37.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:37 smithi066 ceph-mon[30259]: osdmap e31: 3 total, 3 up, 3 in 2024-03-26T01:58:37.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:37 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:37.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:37 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:37.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:37 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:38.019 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:37 smithi055 ceph-mon[30619]: pgmap v103: 129 pgs: 64 unknown, 20 creating+peering, 45 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-26T01:58:38.019 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:37 smithi055 ceph-mon[30619]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi055=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:38.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:37 smithi055 ceph-mon[30619]: Saving service grafana spec with placement smithi055=a;count:1 2024-03-26T01:58:38.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:37 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:38.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:37 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/2761794246' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T01:58:38.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:37 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/1352153755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T01:58:38.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:37 smithi055 ceph-mon[30619]: osdmap e31: 3 total, 3 up, 3 in 2024-03-26T01:58:38.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:37 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:38.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:37 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:38.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:37 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:38.611 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:38 smithi003 ceph-mon[25810]: Deploying daemon prometheus.a on smithi055 2024-03-26T01:58:38.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:38 smithi066 ceph-mon[30259]: Deploying daemon prometheus.a on smithi055 2024-03-26T01:58:39.019 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:38 smithi055 ceph-mon[30619]: Deploying daemon prometheus.a on smithi055 2024-03-26T01:58:39.444 INFO:teuthology.orchestra.run.smithi003.stdout:[client.0] 2024-03-26T01:58:39.445 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQDPKwJmiWt0GhAAJYT1NUY+6Y4sVJxSDTW+0g== 2024-03-26T01:58:39.750 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:39 smithi003 ceph-mon[25810]: pgmap v106: 129 pgs: 129 active+clean; 579 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 3.2 KiB/s wr, 152 op/s 2024-03-26T01:58:39.750 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:39 smithi003 ceph-mon[25810]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T01:58:39.750 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:39 smithi003 ceph-mon[25810]: Cluster is now healthy 2024-03-26T01:58:39.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:39 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/3818738073' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T01:58:39.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:39 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/3818738073' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T01:58:39.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:39 smithi066 ceph-mon[30259]: pgmap v106: 129 pgs: 129 active+clean; 579 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 3.2 KiB/s wr, 152 op/s 2024-03-26T01:58:39.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:39 smithi066 ceph-mon[30259]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T01:58:39.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:39 smithi066 ceph-mon[30259]: Cluster is now healthy 2024-03-26T01:58:39.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:39 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/3818738073' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T01:58:39.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:39 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/3818738073' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T01:58:40.019 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:39 smithi055 ceph-mon[30619]: pgmap v106: 129 pgs: 129 active+clean; 579 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 3.2 KiB/s wr, 152 op/s 2024-03-26T01:58:40.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:39 smithi055 ceph-mon[30619]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T01:58:40.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:39 smithi055 ceph-mon[30619]: Cluster is now healthy 2024-03-26T01:58:40.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:39 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/3818738073' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T01:58:40.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:39 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/3818738073' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T01:58:40.060 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-26T01:58:40.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-26T01:58:40.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-26T01:58:40.104 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T01:58:40.357 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T01:58:41.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:41 smithi055 ceph-mon[30619]: pgmap v107: 129 pgs: 129 active+clean; 579 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 47 KiB/s rd, 2.2 KiB/s wr, 104 op/s 2024-03-26T01:58:41.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:41 smithi003 ceph-mon[25810]: pgmap v107: 129 pgs: 129 active+clean; 579 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 47 KiB/s rd, 2.2 KiB/s wr, 104 op/s 2024-03-26T01:58:41.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:41 smithi066 ceph-mon[30259]: pgmap v107: 129 pgs: 129 active+clean; 579 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 47 KiB/s rd, 2.2 KiB/s wr, 104 op/s 2024-03-26T01:58:43.033 INFO:teuthology.orchestra.run.smithi055.stdout:[client.1] 2024-03-26T01:58:43.033 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQDTKwJmy6qwARAAR/n9/T64Phsa/isn6xy10A== 2024-03-26T01:58:43.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:43 smithi055 ceph-mon[30619]: pgmap v108: 129 pgs: 129 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 71 KiB/s rd, 3.4 KiB/s wr, 160 op/s 2024-03-26T01:58:43.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:43 smithi055 ceph-mon[30619]: from='client.? 172.21.15.55:0/3192846295' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T01:58:43.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:43 smithi055 ceph-mon[30619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T01:58:43.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:43 smithi055 ceph-mon[30619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T01:58:43.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:43 smithi003 ceph-mon[25810]: pgmap v108: 129 pgs: 129 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 71 KiB/s rd, 3.4 KiB/s wr, 160 op/s 2024-03-26T01:58:43.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:43 smithi003 ceph-mon[25810]: from='client.? 172.21.15.55:0/3192846295' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T01:58:43.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:43 smithi003 ceph-mon[25810]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T01:58:43.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:43 smithi003 ceph-mon[25810]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T01:58:43.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:43 smithi066 ceph-mon[30259]: pgmap v108: 129 pgs: 129 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 71 KiB/s rd, 3.4 KiB/s wr, 160 op/s 2024-03-26T01:58:43.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:43 smithi066 ceph-mon[30259]: from='client.? 172.21.15.55:0/3192846295' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T01:58:43.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:43 smithi066 ceph-mon[30259]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T01:58:43.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:43 smithi066 ceph-mon[30259]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T01:58:44.660 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T01:58:44.660 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-26T01:58:44.660 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-26T01:58:44.702 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T01:58:44.960 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T01:58:45.711 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:45 smithi066 ceph-mon[30259]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 90 KiB/s rd, 4.5 KiB/s wr, 200 op/s 2024-03-26T01:58:45.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:45 smithi003 ceph-mon[25810]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 90 KiB/s rd, 4.5 KiB/s wr, 200 op/s 2024-03-26T01:58:46.019 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:45 smithi055 ceph-mon[30619]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 90 KiB/s rd, 4.5 KiB/s wr, 200 op/s 2024-03-26T01:58:46.802 INFO:teuthology.orchestra.run.smithi066.stdout:[client.2] 2024-03-26T01:58:46.802 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQDWKwJmRY+cLxAAQ5ceWfphGArAJ2920wK3Rw== 2024-03-26T01:58:47.359 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T01:58:47.359 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-03-26T01:58:47.359 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-26T01:58:47.408 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-26T01:58:47.408 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-26T01:58:47.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-26T01:58:47.659 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:58:47.662 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:47 smithi055 ceph-mon[30619]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 73 KiB/s rd, 3.7 KiB/s wr, 163 op/s 2024-03-26T01:58:47.662 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:47 smithi055 ceph-mon[30619]: from='client.? 172.21.15.66:0/2791305159' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T01:58:47.662 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:47 smithi055 ceph-mon[30619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T01:58:47.662 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:47 smithi055 ceph-mon[30619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T01:58:47.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:47 smithi066 ceph-mon[30259]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 73 KiB/s rd, 3.7 KiB/s wr, 163 op/s 2024-03-26T01:58:47.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:47 smithi066 ceph-mon[30259]: from='client.? 172.21.15.66:0/2791305159' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T01:58:47.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:47 smithi066 ceph-mon[30259]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T01:58:47.728 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:47 smithi066 ceph-mon[30259]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T01:58:47.781 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:47 smithi003 ceph-mon[25810]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 73 KiB/s rd, 3.7 KiB/s wr, 163 op/s 2024-03-26T01:58:47.781 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:47 smithi003 ceph-mon[25810]: from='client.? 172.21.15.66:0/2791305159' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T01:58:47.781 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:47 smithi003 ceph-mon[25810]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T01:58:47.781 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:47 smithi003 ceph-mon[25810]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T01:58:49.511 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:58:49.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:49 smithi055 ceph-mon[30619]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 66 KiB/s rd, 3.2 KiB/s wr, 147 op/s 2024-03-26T01:58:49.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:49 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/1605785894' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T01:58:49.778 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:49 smithi003 ceph-mon[25810]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 66 KiB/s rd, 3.2 KiB/s wr, 147 op/s 2024-03-26T01:58:49.778 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:49 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/1605785894' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T01:58:49.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:49 smithi066 ceph-mon[30259]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 66 KiB/s rd, 3.2 KiB/s wr, 147 op/s 2024-03-26T01:58:49.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:49 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/1605785894' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T01:58:50.141 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":16,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":2698329634},{"type":"v1","addr":"172.21.15.3:6801","nonce":2698329634}]},"active_addr":"172.21.15.3:6801/2698329634","active_change":"2024-03-26T01:55:47.993192+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.3:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":3036337529}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":3847987293}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":2188787127}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":2008756594}]}]} 2024-03-26T01:58:50.148 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-26T01:58:50.148 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T01:58:50.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T01:58:50.387 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:58:50.520 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 systemd[1]: Starting Ceph prometheus.a for b95eae18-eb13-11ee-95d0-87774f69a715... 2024-03-26T01:58:50.847 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 podman[41768]: 2024-03-26 01:58:50.497659856 +0000 UTC m=+0.018534120 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-26T01:58:50.848 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 podman[41768]: 2024-03-26 01:58:50.600698485 +0000 UTC m=+0.121572735 container create d99a45fedd93fa24f72145ddf17866204bf1d09d401ceea119b00fe67d3de245 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T01:58:50.848 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 podman[41768]: 2024-03-26 01:58:50.794462232 +0000 UTC m=+0.315336482 container init d99a45fedd93fa24f72145ddf17866204bf1d09d401ceea119b00fe67d3de245 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T01:58:50.848 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 podman[41768]: 2024-03-26 01:58:50.800215465 +0000 UTC m=+0.321089715 container start d99a45fedd93fa24f72145ddf17866204bf1d09d401ceea119b00fe67d3de245 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T01:58:50.848 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:58:50.823Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T01:58:50.848 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:58:50.823Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T01:58:50.848 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:58:50.823Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi055 (none))" 2024-03-26T01:58:50.848 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:58:50.823Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T01:58:50.848 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:58:50.823Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T01:58:50.848 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:58:50.824Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T01:58:50.848 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:58:50.825Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T01:58:50.848 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:58:50.827Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T01:58:50.849 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:58:50.827Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T01:58:50.849 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:58:50.829Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T01:58:50.849 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:58:50.829Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.04µs 2024-03-26T01:58:50.849 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:58:50.829Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T01:58:50.849 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:58:50.829Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-26T01:58:50.849 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:58:50.829Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=27.77µs wal_replay_duration=200.149µs wbl_replay_duration=132ns total_replay_duration=256.879µs 2024-03-26T01:58:50.849 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:58:50.830Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T01:58:50.849 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:58:50.830Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T01:58:50.849 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:58:50.830Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T01:58:50.849 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:58:50.847Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.807571ms db_storage=679ns remote_storage=1.111µs web_handler=380ns query_engine=494ns scrape=587.863µs scrape_sd=66.376µs notify=672ns notify_sd=1.304µs rules=16.969266ms tracing=6.361µs 2024-03-26T01:58:50.849 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:58:50.847Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T01:58:50.849 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:58:50.847Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T01:58:51.269 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 bash[41768]: d99a45fedd93fa24f72145ddf17866204bf1d09d401ceea119b00fe67d3de245 2024-03-26T01:58:51.269 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:58:50 smithi055 systemd[1]: Started Ceph prometheus.a for b95eae18-eb13-11ee-95d0-87774f69a715. 2024-03-26T01:58:52.007 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:51 smithi003 ceph-mon[25810]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 37 KiB/s rd, 1.9 KiB/s wr, 83 op/s 2024-03-26T01:58:52.007 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:52.008 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:52.008 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:52.008 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:51 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T01:58:52.197 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:58:52.197 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":31,"fsid":"b95eae18-eb13-11ee-95d0-87774f69a715","created":"2024-03-26T01:54:47.555614+0000","modified":"2024-03-26T01:58:37.170831+0000","last_up_change":"2024-03-26T01:58:16.536620+0000","last_in_change":"2024-03-26T01:57:57.288513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T01:58:18.340237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-26T01:58:28.774186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-26T01:58:30.452669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-26T01:58:32.404917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-26T01:58:34.478852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b217e513-44cb-41f3-a109-5a3688f2c8ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":388847396},{"type":"v1","addr":"172.21.15.3:6803","nonce":388847396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":388847396},{"type":"v1","addr":"172.21.15.3:6805","nonce":388847396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":388847396},{"type":"v1","addr":"172.21.15.3:6809","nonce":388847396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":388847396},{"type":"v1","addr":"172.21.15.3:6807","nonce":388847396}]},"public_addr":"172.21.15.3:6803/388847396","cluster_addr":"172.21.15.3:6805/388847396","heartbeat_back_addr":"172.21.15.3:6809/388847396","heartbeat_front_addr":"172.21.15.3:6807/388847396","state":["exists","up"]},{"osd":1,"uuid":"5ce006b9-8026-4e11-b436-a97e9453b8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":3045481157},{"type":"v1","addr":"172.21.15.55:6801","nonce":3045481157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":3045481157},{"type":"v1","addr":"172.21.15.55:6803","nonce":3045481157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":3045481157},{"type":"v1","addr":"172.21.15.55:6807","nonce":3045481157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":3045481157},{"type":"v1","addr":"172.21.15.55:6805","nonce":3045481157}]},"public_addr":"172.21.15.55:6801/3045481157","cluster_addr":"172.21.15.55:6803/3045481157","heartbeat_back_addr":"172.21.15.55:6807/3045481157","heartbeat_front_addr":"172.21.15.55:6805/3045481157","state":["exists","up"]},{"osd":2,"uuid":"2d9d6781-7a40-43f3-9068-bd0e4038b318","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":1716775083},{"type":"v1","addr":"172.21.15.66:6801","nonce":1716775083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":1716775083},{"type":"v1","addr":"172.21.15.66:6803","nonce":1716775083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":1716775083},{"type":"v1","addr":"172.21.15.66:6807","nonce":1716775083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":1716775083},{"type":"v1","addr":"172.21.15.66:6805","nonce":1716775083}]},"public_addr":"172.21.15.66:6801/1716775083","cluster_addr":"172.21.15.66:6803/1716775083","heartbeat_back_addr":"172.21.15.66:6807/1716775083","heartbeat_front_addr":"172.21.15.66:6805/1716775083","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:57:25.422890+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:57:49.555207+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:58:14.030714+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/972878027":"2024-03-27T01:55:47.993042+0000","172.21.15.3:6800/372539062":"2024-03-27T01:55:47.993042+0000","172.21.15.3:0/3566218324":"2024-03-27T01:55:20.887332+0000","172.21.15.3:6800/1304425570":"2024-03-27T01:55:20.887332+0000","172.21.15.3:6801/372539062":"2024-03-27T01:55:47.993042+0000","172.21.15.3:6801/1304425570":"2024-03-27T01:55:20.887332+0000","172.21.15.3:0/2863586814":"2024-03-27T01:55:47.993042+0000","172.21.15.3:0/2403034937":"2024-03-27T01:55:20.887332+0000","172.21.15.3:0/949463668":"2024-03-27T01:55:20.887332+0000","172.21.15.3:0/2144919668":"2024-03-27T01:55:47.993042+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T01:58:52.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:51 smithi066 ceph-mon[30259]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 37 KiB/s rd, 1.9 KiB/s wr, 83 op/s 2024-03-26T01:58:52.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:51 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:52.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:51 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:52.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:51 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:52.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:51 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T01:58:52.228 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:52 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: ignoring --setuser ceph since I am not root 2024-03-26T01:58:52.228 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:52 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: ignoring --setgroup ceph since I am not root 2024-03-26T01:58:52.228 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:52 smithi066 ceph-mgr[32384]: -- 172.21.15.66:0/392740997 <== mon.2 v2:172.21.15.55:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cdd9fb61a0 con 0x55cdd9f8d000 2024-03-26T01:58:52.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:51 smithi055 ceph-mon[30619]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 37 KiB/s rd, 1.9 KiB/s wr, 83 op/s 2024-03-26T01:58:52.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:51 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:52.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:51 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:52.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:51 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' 2024-03-26T01:58:52.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:51 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T01:58:52.270 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:52 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: ignoring --setuser ceph since I am not root 2024-03-26T01:58:52.270 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:52 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: ignoring --setgroup ceph since I am not root 2024-03-26T01:58:52.270 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:52 smithi055 ceph-mgr[32103]: -- 172.21.15.55:0/2650407710 <== mon.2 v2:172.21.15.55:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556882d341a0 con 0x556882d0b000 2024-03-26T01:58:52.390 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:52 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: ignoring --setuser ceph since I am not root 2024-03-26T01:58:52.391 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:52 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: ignoring --setgroup ceph since I am not root 2024-03-26T01:58:52.655 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:52 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:52.559+0000 7f9227e86200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T01:58:52.655 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:52 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:52.654+0000 7f9227e86200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T01:58:52.824 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T01:58:52.824 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T01:58:52.976 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:52 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:52.566+0000 7f1eaf368200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T01:58:52.976 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:52 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:52.662+0000 7f1eaf368200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T01:58:52.980 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:52 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:52.559+0000 7f38e1c1b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T01:58:52.980 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:52 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:52.654+0000 7f38e1c1b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T01:58:53.067 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:58:53.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:52 smithi066 ceph-mon[30259]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T01:58:53.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:52 smithi066 ceph-mon[30259]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-26T01:58:53.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:52 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/286093444' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T01:58:53.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:52 smithi055 ceph-mon[30619]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T01:58:53.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:52 smithi055 ceph-mon[30619]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-26T01:58:53.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:52 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/286093444' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T01:58:53.290 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:52 smithi003 ceph-mon[25810]: from='mgr.14150 172.21.15.3:0/4213910422' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T01:58:53.290 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:52 smithi003 ceph-mon[25810]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-26T01:58:53.290 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:52 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/286093444' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T01:58:53.562 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:53 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:53.332+0000 7f9227e86200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T01:58:53.563 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:53 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:53.427+0000 7f9227e86200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T01:58:53.645 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:53 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:53.309+0000 7f38e1c1b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T01:58:53.645 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:53 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:53.404+0000 7f38e1c1b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T01:58:53.645 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:53 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:53.547+0000 7f38e1c1b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T01:58:53.645 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:53 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:53.645+0000 7f38e1c1b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T01:58:53.666 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:53 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:53.324+0000 7f1eaf368200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T01:58:53.666 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:53 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:53.420+0000 7f1eaf368200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T01:58:53.666 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:53 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:53.565+0000 7f1eaf368200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T01:58:53.666 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:53 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:53.665+0000 7f1eaf368200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T01:58:53.890 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:53 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:53.578+0000 7f9227e86200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T01:58:53.890 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:53 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:53.676+0000 7f9227e86200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T01:58:53.891 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:53 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:53.761+0000 7f9227e86200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T01:58:53.977 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:53 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:53.751+0000 7f1eaf368200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T01:58:53.998 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:53 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:53.730+0000 7f38e1c1b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T01:58:53.998 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:53 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:53.997+0000 7f38e1c1b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T01:58:54.269 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:54 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:54.110+0000 7f38e1c1b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T01:58:54.269 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:54 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:54.199+0000 7f38e1c1b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T01:58:54.324 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:54 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:54.036+0000 7f9227e86200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T01:58:54.324 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:54 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:54.149+0000 7f9227e86200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T01:58:54.325 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:54 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:54.238+0000 7f9227e86200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T01:58:54.327 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:54 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:54.024+0000 7f1eaf368200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T01:58:54.327 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:54 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:54.138+0000 7f1eaf368200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T01:58:54.327 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:54 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:54.228+0000 7f1eaf368200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T01:58:54.578 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:54 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:54.336+0000 7f9227e86200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T01:58:54.578 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:54 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:54.520+0000 7f9227e86200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T01:58:54.727 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:54 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:54.326+0000 7f1eaf368200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T01:58:54.727 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:54 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:54.508+0000 7f1eaf368200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T01:58:54.769 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:54 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:54.296+0000 7f38e1c1b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T01:58:54.769 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:54 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:54.474+0000 7f38e1c1b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T01:58:55.090 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:58:55.090 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":31,"fsid":"b95eae18-eb13-11ee-95d0-87774f69a715","created":"2024-03-26T01:54:47.555614+0000","modified":"2024-03-26T01:58:37.170831+0000","last_up_change":"2024-03-26T01:58:16.536620+0000","last_in_change":"2024-03-26T01:57:57.288513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T01:58:18.340237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-26T01:58:28.774186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-26T01:58:30.452669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-26T01:58:32.404917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-26T01:58:34.478852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b217e513-44cb-41f3-a109-5a3688f2c8ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":388847396},{"type":"v1","addr":"172.21.15.3:6803","nonce":388847396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":388847396},{"type":"v1","addr":"172.21.15.3:6805","nonce":388847396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":388847396},{"type":"v1","addr":"172.21.15.3:6809","nonce":388847396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":388847396},{"type":"v1","addr":"172.21.15.3:6807","nonce":388847396}]},"public_addr":"172.21.15.3:6803/388847396","cluster_addr":"172.21.15.3:6805/388847396","heartbeat_back_addr":"172.21.15.3:6809/388847396","heartbeat_front_addr":"172.21.15.3:6807/388847396","state":["exists","up"]},{"osd":1,"uuid":"5ce006b9-8026-4e11-b436-a97e9453b8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":3045481157},{"type":"v1","addr":"172.21.15.55:6801","nonce":3045481157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":3045481157},{"type":"v1","addr":"172.21.15.55:6803","nonce":3045481157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":3045481157},{"type":"v1","addr":"172.21.15.55:6807","nonce":3045481157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":3045481157},{"type":"v1","addr":"172.21.15.55:6805","nonce":3045481157}]},"public_addr":"172.21.15.55:6801/3045481157","cluster_addr":"172.21.15.55:6803/3045481157","heartbeat_back_addr":"172.21.15.55:6807/3045481157","heartbeat_front_addr":"172.21.15.55:6805/3045481157","state":["exists","up"]},{"osd":2,"uuid":"2d9d6781-7a40-43f3-9068-bd0e4038b318","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":1716775083},{"type":"v1","addr":"172.21.15.66:6801","nonce":1716775083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":1716775083},{"type":"v1","addr":"172.21.15.66:6803","nonce":1716775083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":1716775083},{"type":"v1","addr":"172.21.15.66:6807","nonce":1716775083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":1716775083},{"type":"v1","addr":"172.21.15.66:6805","nonce":1716775083}]},"public_addr":"172.21.15.66:6801/1716775083","cluster_addr":"172.21.15.66:6803/1716775083","heartbeat_back_addr":"172.21.15.66:6807/1716775083","heartbeat_front_addr":"172.21.15.66:6805/1716775083","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:57:25.422890+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:57:49.555207+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:58:14.030714+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/972878027":"2024-03-27T01:55:47.993042+0000","172.21.15.3:6800/372539062":"2024-03-27T01:55:47.993042+0000","172.21.15.3:0/3566218324":"2024-03-27T01:55:20.887332+0000","172.21.15.3:6800/1304425570":"2024-03-27T01:55:20.887332+0000","172.21.15.3:6801/372539062":"2024-03-27T01:55:47.993042+0000","172.21.15.3:6801/1304425570":"2024-03-27T01:55:20.887332+0000","172.21.15.3:0/2863586814":"2024-03-27T01:55:47.993042+0000","172.21.15.3:0/2403034937":"2024-03-27T01:55:20.887332+0000","172.21.15.3:0/949463668":"2024-03-27T01:55:20.887332+0000","172.21.15.3:0/2144919668":"2024-03-27T01:55:47.993042+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T01:58:55.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:55 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/1208740898' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T01:58:55.462 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:55 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/1208740898' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T01:58:55.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:55 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/1208740898' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T01:58:55.672 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:55 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:55.548+0000 7f9227e86200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T01:58:55.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T01:58:55.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T01:58:55.702 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T01:58:55.771 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:55 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:55.461+0000 7f38e1c1b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T01:58:55.771 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:55 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:55.595+0000 7f38e1c1b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T01:58:55.977 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:55 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:55.523+0000 7f1eaf368200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T01:58:55.977 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:55 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:55.658+0000 7f1eaf368200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T01:58:55.978 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:58:55.978 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:58:55.979 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:58:56.075 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:55 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:55.684+0000 7f9227e86200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T01:58:56.227 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:55 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:55.934+0000 7f38e1c1b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T01:58:56.227 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:56 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:56.025+0000 7f38e1c1b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T01:58:56.227 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:56 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:56.226+0000 7f38e1c1b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:58:56.302 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:56 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:56.003+0000 7f1eaf368200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T01:58:56.302 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:56 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:56.096+0000 7f1eaf368200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T01:58:56.302 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:56 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:56.301+0000 7f1eaf368200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:58:56.376 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:56 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:56.074+0000 7f9227e86200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T01:58:56.376 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:56 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:56.172+0000 7f9227e86200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T01:58:56.519 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:56 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:56.417+0000 7f38e1c1b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T01:58:56.640 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:56 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:56.382+0000 7f9227e86200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:58:56.640 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:56 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:56.590+0000 7f9227e86200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T01:58:56.727 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:56 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:56.491+0000 7f1eaf368200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T01:58:56.970 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:56 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:56.709+0000 7f38e1c1b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:58:57.038 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:56 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:56.892+0000 7f9227e86200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:58:57.056 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:56 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:56.787+0000 7f1eaf368200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:58:57.056 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:57 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:57.055+0000 7f1eaf368200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T01:58:57.269 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:56 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:56.970+0000 7f38e1c1b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T01:58:57.269 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:57 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:57.212+0000 7f38e1c1b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T01:58:57.340 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:57 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:57.157+0000 7f9227e86200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T01:58:57.477 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:57 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:57.299+0000 7f1eaf368200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T01:58:57.640 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:57 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:57.401+0000 7f9227e86200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T01:58:58.390 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:58 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:58.239+0000 7f9227e86200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T01:58:58.477 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:58 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:58.134+0000 7f1eaf368200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T01:58:58.519 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:58 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:58.037+0000 7f38e1c1b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T01:58:58.941 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:58 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:58.619+0000 7f38e1c1b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T01:58:58.941 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:58 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:58.719+0000 7f38e1c1b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T01:58:58.941 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:58 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:58.940+0000 7f38e1c1b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T01:58:58.981 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:58 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:58.837+0000 7f9227e86200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T01:58:58.981 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:58 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:58.939+0000 7f9227e86200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T01:58:59.017 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:58 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:58.726+0000 7f1eaf368200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T01:58:59.017 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:58 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:58.828+0000 7f1eaf368200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T01:58:59.267 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:59 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:59.172+0000 7f9227e86200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T01:58:59.267 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:59 smithi003 ceph-mon[25810]: Standby manager daemon x restarted 2024-03-26T01:58:59.267 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:59 smithi003 ceph-mon[25810]: Standby manager daemon x started 2024-03-26T01:58:59.267 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:59 smithi003 ceph-mon[25810]: from='mgr.? 172.21.15.55:0/3169708044' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T01:58:59.268 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:59 smithi003 ceph-mon[25810]: from='mgr.? 172.21.15.55:0/3169708044' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:58:59.268 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:59 smithi003 ceph-mon[25810]: from='mgr.? 172.21.15.55:0/3169708044' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T01:58:59.268 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:58:59 smithi003 ceph-mon[25810]: from='mgr.? 172.21.15.55:0/3169708044' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:58:59.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:59 smithi055 ceph-mon[30619]: Standby manager daemon x restarted 2024-03-26T01:58:59.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:59 smithi055 ceph-mon[30619]: Standby manager daemon x started 2024-03-26T01:58:59.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:59 smithi055 ceph-mon[30619]: from='mgr.? 172.21.15.55:0/3169708044' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T01:58:59.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:59 smithi055 ceph-mon[30619]: from='mgr.? 172.21.15.55:0/3169708044' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:58:59.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:59 smithi055 ceph-mon[30619]: from='mgr.? 172.21.15.55:0/3169708044' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T01:58:59.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:58:59 smithi055 ceph-mon[30619]: from='mgr.? 172.21.15.55:0/3169708044' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:58:59.270 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:58 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: [26/Mar/2024:01:58:58] ENGINE Bus STARTING 2024-03-26T01:58:59.270 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:59 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: CherryPy Checker: 2024-03-26T01:58:59.270 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:59 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: The Application mounted at '' has an empty config. 2024-03-26T01:58:59.270 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:59 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: 2024-03-26T01:58:59.270 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:59 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: [26/Mar/2024:01:58:59] ENGINE Serving on http://:::9283 2024-03-26T01:58:59.270 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:58:59 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x[32080]: [26/Mar/2024:01:58:59] ENGINE Bus STARTED 2024-03-26T01:58:59.328 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:59 smithi066 ceph-mon[30259]: Standby manager daemon x restarted 2024-03-26T01:58:59.328 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:59 smithi066 ceph-mon[30259]: Standby manager daemon x started 2024-03-26T01:58:59.328 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:59 smithi066 ceph-mon[30259]: from='mgr.? 172.21.15.55:0/3169708044' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T01:58:59.328 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:59 smithi066 ceph-mon[30259]: from='mgr.? 172.21.15.55:0/3169708044' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:58:59.328 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:59 smithi066 ceph-mon[30259]: from='mgr.? 172.21.15.55:0/3169708044' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T01:58:59.328 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:58:59 smithi066 ceph-mon[30259]: from='mgr.? 172.21.15.55:0/3169708044' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:58:59.329 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:59 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:59.056+0000 7f1eaf368200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T01:58:59.329 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:59 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: [26/Mar/2024:01:58:59] ENGINE Bus STARTING 2024-03-26T01:58:59.329 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:59 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: CherryPy Checker: 2024-03-26T01:58:59.329 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:59 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: The Application mounted at '' has an empty config. 2024-03-26T01:58:59.329 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:59 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: 2024-03-26T01:58:59.329 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:59 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: [26/Mar/2024:01:58:59] ENGINE Serving on http://:::9283 2024-03-26T01:58:59.329 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:58:59 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z[32361]: [26/Mar/2024:01:58:59] ENGINE Bus STARTED 2024-03-26T01:58:59.622 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:59 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: [26/Mar/2024:01:58:59] ENGINE Bus STARTING 2024-03-26T01:58:59.622 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:59 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: CherryPy Checker: 2024-03-26T01:58:59.622 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:59 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: The Application mounted at '' has an empty config. 2024-03-26T01:58:59.622 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:59 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: 2024-03-26T01:58:59.622 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:59 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: [26/Mar/2024:01:58:59] ENGINE Serving on http://:::9283 2024-03-26T01:58:59.622 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:58:59 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: [26/Mar/2024:01:58:59] ENGINE Bus STARTED 2024-03-26T01:59:00.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: mgrmap e18: y(active, since 3m), standbys: z, x 2024-03-26T01:59:00.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: Standby manager daemon z restarted 2024-03-26T01:59:00.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: Standby manager daemon z started 2024-03-26T01:59:00.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.? 172.21.15.66:0/93955948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T01:59:00.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.? 172.21.15.66:0/93955948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:59:00.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.? 172.21.15.66:0/93955948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T01:59:00.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.? 172.21.15.66:0/93955948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:59:00.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: Active manager daemon y restarted 2024-03-26T01:59:00.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: Activating manager daemon y 2024-03-26T01:59:00.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: osdmap e32: 3 total, 3 up, 3 in 2024-03-26T01:59:00.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: mgrmap e19: y(active, starting, since 0.0111079s), standbys: z, x 2024-03-26T01:59:00.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:59:00.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:59:00.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:59:00.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T01:59:00.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T01:59:00.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T01:59:00.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:59:00.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:59:00.143 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:59:00.143 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T01:59:00.143 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T01:59:00.143 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T01:59:00.143 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: Manager daemon y is now available 2024-03-26T01:59:00.143 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:00.143 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:00.143 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:00.143 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:59:00.143 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:59:00.143 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:00 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T01:59:00.355 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: mgrmap e18: y(active, since 3m), standbys: z, x 2024-03-26T01:59:00.355 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: Standby manager daemon z restarted 2024-03-26T01:59:00.355 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: Standby manager daemon z started 2024-03-26T01:59:00.356 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.? 172.21.15.66:0/93955948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T01:59:00.356 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.? 172.21.15.66:0/93955948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:59:00.356 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.? 172.21.15.66:0/93955948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T01:59:00.356 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.? 172.21.15.66:0/93955948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:59:00.356 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: Active manager daemon y restarted 2024-03-26T01:59:00.356 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: Activating manager daemon y 2024-03-26T01:59:00.357 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: osdmap e32: 3 total, 3 up, 3 in 2024-03-26T01:59:00.357 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: mgrmap e19: y(active, starting, since 0.0111079s), standbys: z, x 2024-03-26T01:59:00.357 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:59:00.357 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:59:00.357 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:59:00.357 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T01:59:00.358 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T01:59:00.358 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T01:59:00.358 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:59:00.358 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:59:00.358 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:59:00.358 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T01:59:00.359 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T01:59:00.359 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T01:59:00.359 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: Manager daemon y is now available 2024-03-26T01:59:00.359 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:00.359 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:00.359 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:00.360 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:59:00.360 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:59:00.360 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:00 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T01:59:00.411 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: mgrmap e18: y(active, since 3m), standbys: z, x 2024-03-26T01:59:00.411 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: Standby manager daemon z restarted 2024-03-26T01:59:00.412 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: Standby manager daemon z started 2024-03-26T01:59:00.412 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.? 172.21.15.66:0/93955948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T01:59:00.412 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.? 172.21.15.66:0/93955948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:59:00.412 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.? 172.21.15.66:0/93955948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T01:59:00.412 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.? 172.21.15.66:0/93955948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:59:00.412 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: Active manager daemon y restarted 2024-03-26T01:59:00.412 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: Activating manager daemon y 2024-03-26T01:59:00.412 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: osdmap e32: 3 total, 3 up, 3 in 2024-03-26T01:59:00.412 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: mgrmap e19: y(active, starting, since 0.0111079s), standbys: z, x 2024-03-26T01:59:00.412 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:59:00.412 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:59:00.412 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:59:00.412 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T01:59:00.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T01:59:00.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T01:59:00.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:59:00.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:59:00.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:59:00.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T01:59:00.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T01:59:00.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T01:59:00.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: Manager daemon y is now available 2024-03-26T01:59:00.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:00.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:00.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:00.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:59:00.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:59:00.414 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:00 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T01:59:00.863 INFO:teuthology.orchestra.run.smithi003.stdout:77309411337 2024-03-26T01:59:00.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T01:59:01.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:01 smithi066 ceph-mon[30259]: [26/Mar/2024:01:59:00] ENGINE Bus STARTING 2024-03-26T01:59:01.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:01 smithi066 ceph-mon[30259]: mgrmap e20: y(active, since 1.01486s), standbys: z, x 2024-03-26T01:59:01.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:01 smithi066 ceph-mon[30259]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:01.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:01 smithi066 ceph-mon[30259]: [26/Mar/2024:01:59:00] ENGINE Serving on http://172.21.15.3:8765 2024-03-26T01:59:01.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:01 smithi066 ceph-mon[30259]: [26/Mar/2024:01:59:00] ENGINE Serving on https://172.21.15.3:7150 2024-03-26T01:59:01.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:01 smithi066 ceph-mon[30259]: [26/Mar/2024:01:59:00] ENGINE Bus STARTED 2024-03-26T01:59:01.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:01 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:01.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:01 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:01.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:01 smithi055 ceph-mon[30619]: [26/Mar/2024:01:59:00] ENGINE Bus STARTING 2024-03-26T01:59:01.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:01 smithi055 ceph-mon[30619]: mgrmap e20: y(active, since 1.01486s), standbys: z, x 2024-03-26T01:59:01.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:01 smithi055 ceph-mon[30619]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:01.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:01 smithi055 ceph-mon[30619]: [26/Mar/2024:01:59:00] ENGINE Serving on http://172.21.15.3:8765 2024-03-26T01:59:01.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:01 smithi055 ceph-mon[30619]: [26/Mar/2024:01:59:00] ENGINE Serving on https://172.21.15.3:7150 2024-03-26T01:59:01.520 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:01 smithi055 ceph-mon[30619]: [26/Mar/2024:01:59:00] ENGINE Bus STARTED 2024-03-26T01:59:01.521 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:01 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:01.521 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:01 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:01.533 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:01 smithi003 ceph-mon[25810]: [26/Mar/2024:01:59:00] ENGINE Bus STARTING 2024-03-26T01:59:01.533 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:01 smithi003 ceph-mon[25810]: mgrmap e20: y(active, since 1.01486s), standbys: z, x 2024-03-26T01:59:01.533 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:01 smithi003 ceph-mon[25810]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:01.533 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:01 smithi003 ceph-mon[25810]: [26/Mar/2024:01:59:00] ENGINE Serving on http://172.21.15.3:8765 2024-03-26T01:59:01.533 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:01 smithi003 ceph-mon[25810]: [26/Mar/2024:01:59:00] ENGINE Serving on https://172.21.15.3:7150 2024-03-26T01:59:01.533 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:01 smithi003 ceph-mon[25810]: [26/Mar/2024:01:59:00] ENGINE Bus STARTED 2024-03-26T01:59:01.533 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:01 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:01.533 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:01 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:01.560 INFO:teuthology.orchestra.run.smithi003.stdout:34359738387 2024-03-26T01:59:01.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T01:59:02.757 INFO:teuthology.orchestra.run.smithi003.stdout:55834574865 2024-03-26T01:59:02.757 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T01:59:02.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:02 smithi066 ceph-mon[30259]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:02.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:02 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:02.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:02 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:02.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:02 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:02.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:02 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:02.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:02 smithi066 ceph-mon[30259]: mgrmap e21: y(active, since 3s), standbys: z, x 2024-03-26T01:59:03.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:02 smithi055 ceph-mon[30619]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:03.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:02 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:03.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:02 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:03.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:02 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:03.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:02 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:03.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:02 smithi055 ceph-mon[30619]: mgrmap e21: y(active, since 3s), standbys: z, x 2024-03-26T01:59:03.069 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:59:03.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:02 smithi003 ceph-mon[25810]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:03.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:02 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:03.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:02 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:03.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:02 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:03.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:02 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:03.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:02 smithi003 ceph-mon[25810]: mgrmap e21: y(active, since 3s), standbys: z, x 2024-03-26T01:59:03.304 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:59:03.785 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:03 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:03.785 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:03 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:03.785 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:03 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:59:03.785 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:03 smithi003 ceph-mon[25810]: Adjusting osd_memory_target on smithi055 to 16069M 2024-03-26T01:59:03.786 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:03 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:03.900 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:59:04.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:03 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:04.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:03 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:04.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:03 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:59:04.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:03 smithi066 ceph-mon[30259]: Adjusting osd_memory_target on smithi055 to 16069M 2024-03-26T01:59:04.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:03 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:04.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:03 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:04.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:03 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:04.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:03 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:59:04.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:03 smithi055 ceph-mon[30619]: Adjusting osd_memory_target on smithi055 to 16069M 2024-03-26T01:59:04.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:03 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:04.996 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:04 smithi003 ceph-mon[25810]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 0 B/s wr, 38 op/s 2024-03-26T01:59:04.996 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:04 smithi003 ceph-mon[25810]: mgrmap e22: y(active, since 4s), standbys: z, x 2024-03-26T01:59:04.996 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:04 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:04.996 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:04 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:05.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:04 smithi066 ceph-mon[30259]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 0 B/s wr, 38 op/s 2024-03-26T01:59:05.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:04 smithi066 ceph-mon[30259]: mgrmap e22: y(active, since 4s), standbys: z, x 2024-03-26T01:59:05.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:04 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:05.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:04 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:05.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:04 smithi055 ceph-mon[30619]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 0 B/s wr, 38 op/s 2024-03-26T01:59:05.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:04 smithi055 ceph-mon[30619]: mgrmap e22: y(active, since 4s), standbys: z, x 2024-03-26T01:59:05.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:04 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:05.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:04 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:05.911 INFO:teuthology.orchestra.run.smithi003.stdout:77309411339 2024-03-26T01:59:06.792 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411337 got 77309411339 for osd.2 2024-03-26T01:59:06.792 DEBUG:teuthology.parallel:result is None 2024-03-26T01:59:07.019 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:06 smithi055 ceph-mon[30619]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-26T01:59:07.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:06 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:07.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:06 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:07.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:06 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:59:07.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:06 smithi055 ceph-mon[30619]: Adjusting osd_memory_target on smithi003 to 16069M 2024-03-26T01:59:07.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:06 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:07.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:06 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:07.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:06 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:59:07.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:06 smithi055 ceph-mon[30619]: Updating smithi003:/etc/ceph/ceph.conf 2024-03-26T01:59:07.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:06 smithi055 ceph-mon[30619]: Updating smithi055:/etc/ceph/ceph.conf 2024-03-26T01:59:07.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:06 smithi055 ceph-mon[30619]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-26T01:59:07.020 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:06 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/2569125550' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T01:59:07.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:06 smithi055 ceph-mon[30619]: Updating smithi003:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:59:07.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:06 smithi055 ceph-mon[30619]: Updating smithi055:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:59:07.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:06 smithi055 ceph-mon[30619]: Updating smithi066:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:59:07.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:06 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:07.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:06 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:07.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:06 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:07.021 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:06 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:07.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:06 smithi003 ceph-mon[25810]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-26T01:59:07.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:06 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:07.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:06 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:07.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:06 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:59:07.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:06 smithi003 ceph-mon[25810]: Adjusting osd_memory_target on smithi003 to 16069M 2024-03-26T01:59:07.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:06 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:07.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:06 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:07.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:06 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:59:07.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:06 smithi003 ceph-mon[25810]: Updating smithi003:/etc/ceph/ceph.conf 2024-03-26T01:59:07.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:06 smithi003 ceph-mon[25810]: Updating smithi055:/etc/ceph/ceph.conf 2024-03-26T01:59:07.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:06 smithi003 ceph-mon[25810]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-26T01:59:07.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:06 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/2569125550' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T01:59:07.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:06 smithi003 ceph-mon[25810]: Updating smithi003:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:59:07.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:06 smithi003 ceph-mon[25810]: Updating smithi055:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:59:07.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:06 smithi003 ceph-mon[25810]: Updating smithi066:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:59:07.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:06 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:07.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:06 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:07.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:06 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:07.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:06 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:07.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:06 smithi066 ceph-mon[30259]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-26T01:59:07.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:06 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:07.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:06 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:07.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:06 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:59:07.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:06 smithi066 ceph-mon[30259]: Adjusting osd_memory_target on smithi003 to 16069M 2024-03-26T01:59:07.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:06 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:07.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:06 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:07.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:06 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:59:07.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:06 smithi066 ceph-mon[30259]: Updating smithi003:/etc/ceph/ceph.conf 2024-03-26T01:59:07.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:06 smithi066 ceph-mon[30259]: Updating smithi055:/etc/ceph/ceph.conf 2024-03-26T01:59:07.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:06 smithi066 ceph-mon[30259]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-26T01:59:07.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:06 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/2569125550' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T01:59:07.229 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:06 smithi066 ceph-mon[30259]: Updating smithi003:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:59:07.229 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:06 smithi066 ceph-mon[30259]: Updating smithi055:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:59:07.229 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:06 smithi066 ceph-mon[30259]: Updating smithi066:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:59:07.229 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:06 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:07.229 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:06 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:07.229 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:06 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:07.229 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:06 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:08.048 INFO:teuthology.orchestra.run.smithi003.stdout:55834574866 2024-03-26T01:59:08.138 INFO:teuthology.orchestra.run.smithi003.stdout:34359738389 2024-03-26T01:59:08.139 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:07 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: ::ffff:172.21.15.55 - - [26/Mar/2024:01:59:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T01:59:08.139 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:07 smithi003 ceph-mon[25810]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:59:08.139 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:07 smithi003 ceph-mon[25810]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:59:08.139 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:07 smithi003 ceph-mon[25810]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:59:08.139 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:07 smithi003 ceph-mon[25810]: Updating smithi003:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:59:08.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:07 smithi003 ceph-mon[25810]: Updating smithi055:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:59:08.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:07 smithi003 ceph-mon[25810]: Updating smithi066:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:59:08.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:07 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:08.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:07 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:08.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:07 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:08.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:07 smithi003 ceph-mon[25810]: Deploying daemon alertmanager.a on smithi003 2024-03-26T01:59:08.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:07 smithi066 ceph-mon[30259]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:59:08.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:07 smithi066 ceph-mon[30259]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:59:08.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:07 smithi066 ceph-mon[30259]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:59:08.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:07 smithi066 ceph-mon[30259]: Updating smithi003:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:59:08.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:07 smithi066 ceph-mon[30259]: Updating smithi055:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:59:08.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:07 smithi066 ceph-mon[30259]: Updating smithi066:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:59:08.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:07 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:08.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:07 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:08.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:07 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:08.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:07 smithi066 ceph-mon[30259]: Deploying daemon alertmanager.a on smithi003 2024-03-26T01:59:08.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:07 smithi055 ceph-mon[30619]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:59:08.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:07 smithi055 ceph-mon[30619]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:59:08.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:07 smithi055 ceph-mon[30619]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:59:08.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:07 smithi055 ceph-mon[30619]: Updating smithi003:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:59:08.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:07 smithi055 ceph-mon[30619]: Updating smithi055:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:59:08.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:07 smithi055 ceph-mon[30619]: Updating smithi066:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:59:08.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:07 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:08.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:07 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:08.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:07 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:08.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:07 smithi055 ceph-mon[30619]: Deploying daemon alertmanager.a on smithi003 2024-03-26T01:59:08.769 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738387 got 34359738389 for osd.0 2024-03-26T01:59:08.769 DEBUG:teuthology.parallel:result is None 2024-03-26T01:59:09.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:08 smithi003 ceph-mon[25810]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T01:59:09.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:08 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/1349802561' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T01:59:09.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:08 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/1956307895' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T01:59:09.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:08 smithi066 ceph-mon[30259]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T01:59:09.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:08 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/1349802561' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T01:59:09.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:08 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/1956307895' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T01:59:09.236 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574865 got 55834574866 for osd.1 2024-03-26T01:59:09.236 DEBUG:teuthology.parallel:result is None 2024-03-26T01:59:09.237 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T01:59:09.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T01:59:09.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:08 smithi055 ceph-mon[30619]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T01:59:09.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:08 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/1349802561' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T01:59:09.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:08 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/1956307895' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T01:59:09.496 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:59:10.977 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:10 smithi066 ceph-mon[30259]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T01:59:11.113 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:10 smithi003 ceph-mon[25810]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T01:59:11.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:10 smithi055 ceph-mon[30619]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T01:59:11.735 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:59:11.738 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-26T01:59:13.023 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-03-26T01:59:11.193201+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":1262,"num_read_kb":1077,"num_write":889,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":870,"ondisk_log_size":870,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96636,"kb_used_data":3108,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126532,"statfs":{"total":287972524032,"available":287873568768,"internally_reserved":0,"allocated":3182592,"data_stored":2109504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.001072"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180531+0000","last_change":"2024-03-26T01:58:36.186063+0000","last_active":"2024-03-26T01:58:59.180531+0000","last_peered":"2024-03-26T01:58:59.180531+0000","last_clean":"2024-03-26T01:58:59.180531+0000","last_became_active":"2024-03-26T01:58:36.184564+0000","last_became_peered":"2024-03-26T01:58:36.184564+0000","last_unstale":"2024-03-26T01:58:59.180531+0000","last_undegraded":"2024-03-26T01:58:59.180531+0000","last_fullsized":"2024-03-26T01:58:59.180531+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:58:40.986608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183574+0000","last_change":"2024-03-26T01:58:34.169468+0000","last_active":"2024-03-26T01:58:59.183574+0000","last_peered":"2024-03-26T01:58:59.183574+0000","last_clean":"2024-03-26T01:58:59.183574+0000","last_became_active":"2024-03-26T01:58:34.169220+0000","last_became_peered":"2024-03-26T01:58:34.169220+0000","last_unstale":"2024-03-26T01:58:59.183574+0000","last_undegraded":"2024-03-26T01:58:59.183574+0000","last_fullsized":"2024-03-26T01:58:59.183574+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:21:02.472833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182555+0000","last_change":"2024-03-26T01:58:30.136433+0000","last_active":"2024-03-26T01:58:59.182555+0000","last_peered":"2024-03-26T01:58:59.182555+0000","last_clean":"2024-03-26T01:58:59.182555+0000","last_became_active":"2024-03-26T01:58:30.136121+0000","last_became_peered":"2024-03-26T01:58:30.136121+0000","last_unstale":"2024-03-26T01:58:59.182555+0000","last_undegraded":"2024-03-26T01:58:59.182555+0000","last_fullsized":"2024-03-26T01:58:59.182555+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:13:26.702227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"31'16","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182913+0000","last_change":"2024-03-26T01:58:32.155738+0000","last_active":"2024-03-26T01:58:59.182913+0000","last_peered":"2024-03-26T01:58:59.182913+0000","last_clean":"2024-03-26T01:58:59.182913+0000","last_became_active":"2024-03-26T01:58:32.154739+0000","last_became_peered":"2024-03-26T01:58:32.154739+0000","last_unstale":"2024-03-26T01:58:59.182913+0000","last_undegraded":"2024-03-26T01:58:59.182913+0000","last_fullsized":"2024-03-26T01:58:59.182913+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:15:16.182003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179615+0000","last_change":"2024-03-26T01:58:36.182482+0000","last_active":"2024-03-26T01:58:59.179615+0000","last_peered":"2024-03-26T01:58:59.179615+0000","last_clean":"2024-03-26T01:58:59.179615+0000","last_became_active":"2024-03-26T01:58:36.182070+0000","last_became_peered":"2024-03-26T01:58:36.182070+0000","last_unstale":"2024-03-26T01:58:59.179615+0000","last_undegraded":"2024-03-26T01:58:59.179615+0000","last_fullsized":"2024-03-26T01:58:59.179615+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:22:30.388786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180379+0000","last_change":"2024-03-26T01:58:34.167613+0000","last_active":"2024-03-26T01:58:59.180379+0000","last_peered":"2024-03-26T01:58:59.180379+0000","last_clean":"2024-03-26T01:58:59.180379+0000","last_became_active":"2024-03-26T01:58:34.166603+0000","last_became_peered":"2024-03-26T01:58:34.166603+0000","last_unstale":"2024-03-26T01:58:59.180379+0000","last_undegraded":"2024-03-26T01:58:59.180379+0000","last_fullsized":"2024-03-26T01:58:59.180379+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:40:17.393707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182555+0000","last_change":"2024-03-26T01:58:30.137133+0000","last_active":"2024-03-26T01:58:59.182555+0000","last_peered":"2024-03-26T01:58:59.182555+0000","last_clean":"2024-03-26T01:58:59.182555+0000","last_became_active":"2024-03-26T01:58:30.136181+0000","last_became_peered":"2024-03-26T01:58:30.136181+0000","last_unstale":"2024-03-26T01:58:59.182555+0000","last_undegraded":"2024-03-26T01:58:59.182555+0000","last_fullsized":"2024-03-26T01:58:59.182555+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:27:18.062383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'17","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182336+0000","last_change":"2024-03-26T01:58:32.157641+0000","last_active":"2024-03-26T01:58:59.182336+0000","last_peered":"2024-03-26T01:58:59.182336+0000","last_clean":"2024-03-26T01:58:59.182336+0000","last_became_active":"2024-03-26T01:58:32.157299+0000","last_became_peered":"2024-03-26T01:58:32.157299+0000","last_unstale":"2024-03-26T01:58:59.182336+0000","last_undegraded":"2024-03-26T01:58:59.182336+0000","last_fullsized":"2024-03-26T01:58:59.182336+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:43:42.453770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183883+0000","last_change":"2024-03-26T01:58:36.183012+0000","last_active":"2024-03-26T01:58:59.183883+0000","last_peered":"2024-03-26T01:58:59.183883+0000","last_clean":"2024-03-26T01:58:59.183883+0000","last_became_active":"2024-03-26T01:58:36.182657+0000","last_became_peered":"2024-03-26T01:58:36.182657+0000","last_unstale":"2024-03-26T01:58:59.183883+0000","last_undegraded":"2024-03-26T01:58:59.183883+0000","last_fullsized":"2024-03-26T01:58:59.183883+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:05:25.331266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180748+0000","last_change":"2024-03-26T01:58:34.165372+0000","last_active":"2024-03-26T01:58:59.180748+0000","last_peered":"2024-03-26T01:58:59.180748+0000","last_clean":"2024-03-26T01:58:59.180748+0000","last_became_active":"2024-03-26T01:58:34.164197+0000","last_became_peered":"2024-03-26T01:58:34.164197+0000","last_unstale":"2024-03-26T01:58:59.180748+0000","last_undegraded":"2024-03-26T01:58:59.180748+0000","last_fullsized":"2024-03-26T01:58:59.180748+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:59:03.909926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.186999+0000","last_change":"2024-03-26T01:58:30.139473+0000","last_active":"2024-03-26T01:58:59.186999+0000","last_peered":"2024-03-26T01:58:59.186999+0000","last_clean":"2024-03-26T01:58:59.186999+0000","last_became_active":"2024-03-26T01:58:30.138715+0000","last_became_peered":"2024-03-26T01:58:30.138715+0000","last_unstale":"2024-03-26T01:58:59.186999+0000","last_undegraded":"2024-03-26T01:58:59.186999+0000","last_fullsized":"2024-03-26T01:58:59.186999+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:09:19.811105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181860+0000","last_change":"2024-03-26T01:58:32.159428+0000","last_active":"2024-03-26T01:58:59.181860+0000","last_peered":"2024-03-26T01:58:59.181860+0000","last_clean":"2024-03-26T01:58:59.181860+0000","last_became_active":"2024-03-26T01:58:32.159223+0000","last_became_peered":"2024-03-26T01:58:32.159223+0000","last_unstale":"2024-03-26T01:58:59.181860+0000","last_undegraded":"2024-03-26T01:58:59.181860+0000","last_fullsized":"2024-03-26T01:58:59.181860+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:28:08.287176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180635+0000","last_change":"2024-03-26T01:58:36.188560+0000","last_active":"2024-03-26T01:58:59.180635+0000","last_peered":"2024-03-26T01:58:59.180635+0000","last_clean":"2024-03-26T01:58:59.180635+0000","last_became_active":"2024-03-26T01:58:36.188193+0000","last_became_peered":"2024-03-26T01:58:36.188193+0000","last_unstale":"2024-03-26T01:58:59.180635+0000","last_undegraded":"2024-03-26T01:58:59.180635+0000","last_fullsized":"2024-03-26T01:58:59.180635+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:04:58.798057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180849+0000","last_change":"2024-03-26T01:58:34.167642+0000","last_active":"2024-03-26T01:58:59.180849+0000","last_peered":"2024-03-26T01:58:59.180849+0000","last_clean":"2024-03-26T01:58:59.180849+0000","last_became_active":"2024-03-26T01:58:34.167342+0000","last_became_peered":"2024-03-26T01:58:34.167342+0000","last_unstale":"2024-03-26T01:58:59.180849+0000","last_undegraded":"2024-03-26T01:58:59.180849+0000","last_fullsized":"2024-03-26T01:58:59.180849+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:07:09.662945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185142+0000","last_change":"2024-03-26T01:58:30.137539+0000","last_active":"2024-03-26T01:58:59.185142+0000","last_peered":"2024-03-26T01:58:59.185142+0000","last_clean":"2024-03-26T01:58:59.185142+0000","last_became_active":"2024-03-26T01:58:30.136866+0000","last_became_peered":"2024-03-26T01:58:30.136866+0000","last_unstale":"2024-03-26T01:58:59.185142+0000","last_undegraded":"2024-03-26T01:58:59.185142+0000","last_fullsized":"2024-03-26T01:58:59.185142+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:07:22.543441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"31'18","reported_seq":58,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.186441+0000","last_change":"2024-03-26T01:58:32.156713+0000","last_active":"2024-03-26T01:58:59.186441+0000","last_peered":"2024-03-26T01:58:59.186441+0000","last_clean":"2024-03-26T01:58:59.186441+0000","last_became_active":"2024-03-26T01:58:32.156303+0000","last_became_peered":"2024-03-26T01:58:32.156303+0000","last_unstale":"2024-03-26T01:58:59.186441+0000","last_undegraded":"2024-03-26T01:58:59.186441+0000","last_fullsized":"2024-03-26T01:58:59.186441+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:55:39.189779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181640+0000","last_change":"2024-03-26T01:58:36.186762+0000","last_active":"2024-03-26T01:58:59.181640+0000","last_peered":"2024-03-26T01:58:59.181640+0000","last_clean":"2024-03-26T01:58:59.181640+0000","last_became_active":"2024-03-26T01:58:36.185062+0000","last_became_peered":"2024-03-26T01:58:36.185062+0000","last_unstale":"2024-03-26T01:58:59.181640+0000","last_undegraded":"2024-03-26T01:58:59.181640+0000","last_fullsized":"2024-03-26T01:58:59.181640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:48:06.899232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185938+0000","last_change":"2024-03-26T01:58:34.165186+0000","last_active":"2024-03-26T01:58:59.185938+0000","last_peered":"2024-03-26T01:58:59.185938+0000","last_clean":"2024-03-26T01:58:59.185938+0000","last_became_active":"2024-03-26T01:58:34.164958+0000","last_became_peered":"2024-03-26T01:58:34.164958+0000","last_unstale":"2024-03-26T01:58:59.185938+0000","last_undegraded":"2024-03-26T01:58:59.185938+0000","last_fullsized":"2024-03-26T01:58:59.185938+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:28:54.247036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185766+0000","last_change":"2024-03-26T01:58:30.136301+0000","last_active":"2024-03-26T01:58:59.185766+0000","last_peered":"2024-03-26T01:58:59.185766+0000","last_clean":"2024-03-26T01:58:59.185766+0000","last_became_active":"2024-03-26T01:58:30.134461+0000","last_became_peered":"2024-03-26T01:58:30.134461+0000","last_unstale":"2024-03-26T01:58:59.185766+0000","last_undegraded":"2024-03-26T01:58:59.185766+0000","last_fullsized":"2024-03-26T01:58:59.185766+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:20:47.835309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"31'21","reported_seq":63,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184841+0000","last_change":"2024-03-26T01:58:32.158431+0000","last_active":"2024-03-26T01:58:59.184841+0000","last_peered":"2024-03-26T01:58:59.184841+0000","last_clean":"2024-03-26T01:58:59.184841+0000","last_became_active":"2024-03-26T01:58:32.158133+0000","last_became_peered":"2024-03-26T01:58:32.158133+0000","last_unstale":"2024-03-26T01:58:59.184841+0000","last_undegraded":"2024-03-26T01:58:59.184841+0000","last_fullsized":"2024-03-26T01:58:59.184841+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:28:27.108716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182341+0000","last_change":"2024-03-26T01:58:36.187900+0000","last_active":"2024-03-26T01:58:59.182341+0000","last_peered":"2024-03-26T01:58:59.182341+0000","last_clean":"2024-03-26T01:58:59.182341+0000","last_became_active":"2024-03-26T01:58:36.187576+0000","last_became_peered":"2024-03-26T01:58:36.187576+0000","last_unstale":"2024-03-26T01:58:59.182341+0000","last_undegraded":"2024-03-26T01:58:59.182341+0000","last_fullsized":"2024-03-26T01:58:59.182341+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:50:30.587849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185452+0000","last_change":"2024-03-26T01:58:34.171690+0000","last_active":"2024-03-26T01:58:59.185452+0000","last_peered":"2024-03-26T01:58:59.185452+0000","last_clean":"2024-03-26T01:58:59.185452+0000","last_became_active":"2024-03-26T01:58:34.171272+0000","last_became_peered":"2024-03-26T01:58:34.171272+0000","last_unstale":"2024-03-26T01:58:59.185452+0000","last_undegraded":"2024-03-26T01:58:59.185452+0000","last_fullsized":"2024-03-26T01:58:59.185452+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:26:12.716480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181154+0000","last_change":"2024-03-26T01:58:30.135343+0000","last_active":"2024-03-26T01:58:59.181154+0000","last_peered":"2024-03-26T01:58:59.181154+0000","last_clean":"2024-03-26T01:58:59.181154+0000","last_became_active":"2024-03-26T01:58:30.134604+0000","last_became_peered":"2024-03-26T01:58:30.134604+0000","last_unstale":"2024-03-26T01:58:59.181154+0000","last_undegraded":"2024-03-26T01:58:59.181154+0000","last_fullsized":"2024-03-26T01:58:59.181154+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:07:12.495786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'11","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185585+0000","last_change":"2024-03-26T01:58:32.157084+0000","last_active":"2024-03-26T01:58:59.185585+0000","last_peered":"2024-03-26T01:58:59.185585+0000","last_clean":"2024-03-26T01:58:59.185585+0000","last_became_active":"2024-03-26T01:58:32.156556+0000","last_became_peered":"2024-03-26T01:58:32.156556+0000","last_unstale":"2024-03-26T01:58:59.185585+0000","last_undegraded":"2024-03-26T01:58:59.185585+0000","last_fullsized":"2024-03-26T01:58:59.185585+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:58:48.935597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181749+0000","last_change":"2024-03-26T01:58:36.187742+0000","last_active":"2024-03-26T01:58:59.181749+0000","last_peered":"2024-03-26T01:58:59.181749+0000","last_clean":"2024-03-26T01:58:59.181749+0000","last_became_active":"2024-03-26T01:58:36.187374+0000","last_became_peered":"2024-03-26T01:58:36.187374+0000","last_unstale":"2024-03-26T01:58:59.181749+0000","last_undegraded":"2024-03-26T01:58:59.181749+0000","last_fullsized":"2024-03-26T01:58:59.181749+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:11:14.940926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"31'8","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:05.186074+0000","last_change":"2024-03-26T01:58:34.171888+0000","last_active":"2024-03-26T01:59:05.186074+0000","last_peered":"2024-03-26T01:59:05.186074+0000","last_clean":"2024-03-26T01:59:05.186074+0000","last_became_active":"2024-03-26T01:58:34.171283+0000","last_became_peered":"2024-03-26T01:58:34.171283+0000","last_unstale":"2024-03-26T01:59:05.186074+0000","last_undegraded":"2024-03-26T01:59:05.186074+0000","last_fullsized":"2024-03-26T01:59:05.186074+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:44:21.909014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179715+0000","last_change":"2024-03-26T01:58:30.136341+0000","last_active":"2024-03-26T01:58:59.179715+0000","last_peered":"2024-03-26T01:58:59.179715+0000","last_clean":"2024-03-26T01:58:59.179715+0000","last_became_active":"2024-03-26T01:58:30.136052+0000","last_became_peered":"2024-03-26T01:58:30.136052+0000","last_unstale":"2024-03-26T01:58:59.179715+0000","last_undegraded":"2024-03-26T01:58:59.179715+0000","last_fullsized":"2024-03-26T01:58:59.179715+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:06:48.642790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'15","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179138+0000","last_change":"2024-03-26T01:58:32.161289+0000","last_active":"2024-03-26T01:58:59.179138+0000","last_peered":"2024-03-26T01:58:59.179138+0000","last_clean":"2024-03-26T01:58:59.179138+0000","last_became_active":"2024-03-26T01:58:32.160902+0000","last_became_peered":"2024-03-26T01:58:32.160902+0000","last_unstale":"2024-03-26T01:58:59.179138+0000","last_undegraded":"2024-03-26T01:58:59.179138+0000","last_fullsized":"2024-03-26T01:58:59.179138+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:01:43.388216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181639+0000","last_change":"2024-03-26T01:58:36.188324+0000","last_active":"2024-03-26T01:58:59.181639+0000","last_peered":"2024-03-26T01:58:59.181639+0000","last_clean":"2024-03-26T01:58:59.181639+0000","last_became_active":"2024-03-26T01:58:36.187820+0000","last_became_peered":"2024-03-26T01:58:36.187820+0000","last_unstale":"2024-03-26T01:58:59.181639+0000","last_undegraded":"2024-03-26T01:58:59.181639+0000","last_fullsized":"2024-03-26T01:58:59.181639+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:40:49.955325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181910+0000","last_change":"2024-03-26T01:58:34.173274+0000","last_active":"2024-03-26T01:58:59.181910+0000","last_peered":"2024-03-26T01:58:59.181910+0000","last_clean":"2024-03-26T01:58:59.181910+0000","last_became_active":"2024-03-26T01:58:34.172532+0000","last_became_peered":"2024-03-26T01:58:34.172532+0000","last_unstale":"2024-03-26T01:58:59.181910+0000","last_undegraded":"2024-03-26T01:58:59.181910+0000","last_fullsized":"2024-03-26T01:58:59.181910+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:17:33.036512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183744+0000","last_change":"2024-03-26T01:58:30.139313+0000","last_active":"2024-03-26T01:58:59.183744+0000","last_peered":"2024-03-26T01:58:59.183744+0000","last_clean":"2024-03-26T01:58:59.183744+0000","last_became_active":"2024-03-26T01:58:30.138808+0000","last_became_peered":"2024-03-26T01:58:30.138808+0000","last_unstale":"2024-03-26T01:58:59.183744+0000","last_undegraded":"2024-03-26T01:58:59.183744+0000","last_fullsized":"2024-03-26T01:58:59.183744+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:06:28.430981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"31'41","reported_seq":98,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179720+0000","last_change":"2024-03-26T01:58:32.155029+0000","last_active":"2024-03-26T01:58:59.179720+0000","last_peered":"2024-03-26T01:58:59.179720+0000","last_clean":"2024-03-26T01:58:59.179720+0000","last_became_active":"2024-03-26T01:58:32.154326+0000","last_became_peered":"2024-03-26T01:58:32.154326+0000","last_unstale":"2024-03-26T01:58:59.179720+0000","last_undegraded":"2024-03-26T01:58:59.179720+0000","last_fullsized":"2024-03-26T01:58:59.179720+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:02:12.005363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180695+0000","last_change":"2024-03-26T01:58:36.185062+0000","last_active":"2024-03-26T01:58:59.180695+0000","last_peered":"2024-03-26T01:58:59.180695+0000","last_clean":"2024-03-26T01:58:59.180695+0000","last_became_active":"2024-03-26T01:58:36.184474+0000","last_became_peered":"2024-03-26T01:58:36.184474+0000","last_unstale":"2024-03-26T01:58:59.180695+0000","last_undegraded":"2024-03-26T01:58:59.180695+0000","last_fullsized":"2024-03-26T01:58:59.180695+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:13:33.577968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179208+0000","last_change":"2024-03-26T01:58:34.171576+0000","last_active":"2024-03-26T01:58:59.179208+0000","last_peered":"2024-03-26T01:58:59.179208+0000","last_clean":"2024-03-26T01:58:59.179208+0000","last_became_active":"2024-03-26T01:58:34.171107+0000","last_became_peered":"2024-03-26T01:58:34.171107+0000","last_unstale":"2024-03-26T01:58:59.179208+0000","last_undegraded":"2024-03-26T01:58:59.179208+0000","last_fullsized":"2024-03-26T01:58:59.179208+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:53:57.800083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183266+0000","last_change":"2024-03-26T01:58:30.140709+0000","last_active":"2024-03-26T01:58:59.183266+0000","last_peered":"2024-03-26T01:58:59.183266+0000","last_clean":"2024-03-26T01:58:59.183266+0000","last_became_active":"2024-03-26T01:58:30.140150+0000","last_became_peered":"2024-03-26T01:58:30.140150+0000","last_unstale":"2024-03-26T01:58:59.183266+0000","last_undegraded":"2024-03-26T01:58:59.183266+0000","last_fullsized":"2024-03-26T01:58:59.183266+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:45:05.542683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"31'21","reported_seq":63,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181998+0000","last_change":"2024-03-26T01:58:32.160893+0000","last_active":"2024-03-26T01:58:59.181998+0000","last_peered":"2024-03-26T01:58:59.181998+0000","last_clean":"2024-03-26T01:58:59.181998+0000","last_became_active":"2024-03-26T01:58:32.160597+0000","last_became_peered":"2024-03-26T01:58:32.160597+0000","last_unstale":"2024-03-26T01:58:59.181998+0000","last_undegraded":"2024-03-26T01:58:59.181998+0000","last_fullsized":"2024-03-26T01:58:59.181998+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:53:14.326937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180279+0000","last_change":"2024-03-26T01:58:36.188464+0000","last_active":"2024-03-26T01:58:59.180279+0000","last_peered":"2024-03-26T01:58:59.180279+0000","last_clean":"2024-03-26T01:58:59.180279+0000","last_became_active":"2024-03-26T01:58:36.187867+0000","last_became_peered":"2024-03-26T01:58:36.187867+0000","last_unstale":"2024-03-26T01:58:59.180279+0000","last_undegraded":"2024-03-26T01:58:59.180279+0000","last_fullsized":"2024-03-26T01:58:59.180279+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:10:34.963271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179427+0000","last_change":"2024-03-26T01:58:34.174164+0000","last_active":"2024-03-26T01:58:59.179427+0000","last_peered":"2024-03-26T01:58:59.179427+0000","last_clean":"2024-03-26T01:58:59.179427+0000","last_became_active":"2024-03-26T01:58:34.172669+0000","last_became_peered":"2024-03-26T01:58:34.172669+0000","last_unstale":"2024-03-26T01:58:59.179427+0000","last_undegraded":"2024-03-26T01:58:59.179427+0000","last_fullsized":"2024-03-26T01:58:59.179427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:30:04.294454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182145+0000","last_change":"2024-03-26T01:58:30.139862+0000","last_active":"2024-03-26T01:58:59.182145+0000","last_peered":"2024-03-26T01:58:59.182145+0000","last_clean":"2024-03-26T01:58:59.182145+0000","last_became_active":"2024-03-26T01:58:30.139467+0000","last_became_peered":"2024-03-26T01:58:30.139467+0000","last_unstale":"2024-03-26T01:58:59.182145+0000","last_undegraded":"2024-03-26T01:58:59.182145+0000","last_fullsized":"2024-03-26T01:58:59.182145+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:31:28.069553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"31'12","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182810+0000","last_change":"2024-03-26T01:58:32.157228+0000","last_active":"2024-03-26T01:58:59.182810+0000","last_peered":"2024-03-26T01:58:59.182810+0000","last_clean":"2024-03-26T01:58:59.182810+0000","last_became_active":"2024-03-26T01:58:32.155976+0000","last_became_peered":"2024-03-26T01:58:32.155976+0000","last_unstale":"2024-03-26T01:58:59.182810+0000","last_undegraded":"2024-03-26T01:58:59.182810+0000","last_fullsized":"2024-03-26T01:58:59.182810+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:04:28.437573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184111+0000","last_change":"2024-03-26T01:58:36.182835+0000","last_active":"2024-03-26T01:58:59.184111+0000","last_peered":"2024-03-26T01:58:59.184111+0000","last_clean":"2024-03-26T01:58:59.184111+0000","last_became_active":"2024-03-26T01:58:36.182517+0000","last_became_peered":"2024-03-26T01:58:36.182517+0000","last_unstale":"2024-03-26T01:58:59.184111+0000","last_undegraded":"2024-03-26T01:58:59.184111+0000","last_fullsized":"2024-03-26T01:58:59.184111+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:27:10.816704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180928+0000","last_change":"2024-03-26T01:58:34.169080+0000","last_active":"2024-03-26T01:58:59.180928+0000","last_peered":"2024-03-26T01:58:59.180928+0000","last_clean":"2024-03-26T01:58:59.180928+0000","last_became_active":"2024-03-26T01:58:34.167922+0000","last_became_peered":"2024-03-26T01:58:34.167922+0000","last_unstale":"2024-03-26T01:58:59.180928+0000","last_undegraded":"2024-03-26T01:58:59.180928+0000","last_fullsized":"2024-03-26T01:58:59.180928+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:47:35.604280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.186414+0000","last_change":"2024-03-26T01:58:30.139157+0000","last_active":"2024-03-26T01:58:59.186414+0000","last_peered":"2024-03-26T01:58:59.186414+0000","last_clean":"2024-03-26T01:58:59.186414+0000","last_became_active":"2024-03-26T01:58:30.137565+0000","last_became_peered":"2024-03-26T01:58:30.137565+0000","last_unstale":"2024-03-26T01:58:59.186414+0000","last_undegraded":"2024-03-26T01:58:59.186414+0000","last_fullsized":"2024-03-26T01:58:59.186414+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:25:29.921875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'24","reported_seq":64,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182832+0000","last_change":"2024-03-26T01:58:32.157693+0000","last_active":"2024-03-26T01:58:59.182832+0000","last_peered":"2024-03-26T01:58:59.182832+0000","last_clean":"2024-03-26T01:58:59.182832+0000","last_became_active":"2024-03-26T01:58:32.157400+0000","last_became_peered":"2024-03-26T01:58:32.157400+0000","last_unstale":"2024-03-26T01:58:59.182832+0000","last_undegraded":"2024-03-26T01:58:59.182832+0000","last_fullsized":"2024-03-26T01:58:59.182832+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:29:19.300415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185010+0000","last_change":"2024-03-26T01:58:36.183640+0000","last_active":"2024-03-26T01:58:59.185010+0000","last_peered":"2024-03-26T01:58:59.185010+0000","last_clean":"2024-03-26T01:58:59.185010+0000","last_became_active":"2024-03-26T01:58:36.182837+0000","last_became_peered":"2024-03-26T01:58:36.182837+0000","last_unstale":"2024-03-26T01:58:59.185010+0000","last_undegraded":"2024-03-26T01:58:59.185010+0000","last_fullsized":"2024-03-26T01:58:59.185010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:53:32.261663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179971+0000","last_change":"2024-03-26T01:58:34.171639+0000","last_active":"2024-03-26T01:58:59.179971+0000","last_peered":"2024-03-26T01:58:59.179971+0000","last_clean":"2024-03-26T01:58:59.179971+0000","last_became_active":"2024-03-26T01:58:34.171196+0000","last_became_peered":"2024-03-26T01:58:34.171196+0000","last_unstale":"2024-03-26T01:58:59.179971+0000","last_undegraded":"2024-03-26T01:58:59.179971+0000","last_fullsized":"2024-03-26T01:58:59.179971+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:59:38.446244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182248+0000","last_change":"2024-03-26T01:58:30.140212+0000","last_active":"2024-03-26T01:58:59.182248+0000","last_peered":"2024-03-26T01:58:59.182248+0000","last_clean":"2024-03-26T01:58:59.182248+0000","last_became_active":"2024-03-26T01:58:30.139663+0000","last_became_peered":"2024-03-26T01:58:30.139663+0000","last_unstale":"2024-03-26T01:58:59.182248+0000","last_undegraded":"2024-03-26T01:58:59.182248+0000","last_fullsized":"2024-03-26T01:58:59.182248+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:09:08.640306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"31'15","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.186169+0000","last_change":"2024-03-26T01:58:32.156307+0000","last_active":"2024-03-26T01:58:59.186169+0000","last_peered":"2024-03-26T01:58:59.186169+0000","last_clean":"2024-03-26T01:58:59.186169+0000","last_became_active":"2024-03-26T01:58:32.155809+0000","last_became_peered":"2024-03-26T01:58:32.155809+0000","last_unstale":"2024-03-26T01:58:59.186169+0000","last_undegraded":"2024-03-26T01:58:59.186169+0000","last_fullsized":"2024-03-26T01:58:59.186169+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:27:47.247203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180971+0000","last_change":"2024-03-26T01:58:36.183063+0000","last_active":"2024-03-26T01:58:59.180971+0000","last_peered":"2024-03-26T01:58:59.180971+0000","last_clean":"2024-03-26T01:58:59.180971+0000","last_became_active":"2024-03-26T01:58:36.182678+0000","last_became_peered":"2024-03-26T01:58:36.182678+0000","last_unstale":"2024-03-26T01:58:59.180971+0000","last_undegraded":"2024-03-26T01:58:59.180971+0000","last_fullsized":"2024-03-26T01:58:59.180971+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:44:45.389150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:05.186074+0000","last_change":"2024-03-26T01:58:34.168622+0000","last_active":"2024-03-26T01:59:05.186074+0000","last_peered":"2024-03-26T01:59:05.186074+0000","last_clean":"2024-03-26T01:59:05.186074+0000","last_became_active":"2024-03-26T01:58:34.168268+0000","last_became_peered":"2024-03-26T01:58:34.168268+0000","last_unstale":"2024-03-26T01:59:05.186074+0000","last_undegraded":"2024-03-26T01:59:05.186074+0000","last_fullsized":"2024-03-26T01:59:05.186074+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:50:32.236929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185492+0000","last_change":"2024-03-26T01:58:30.134776+0000","last_active":"2024-03-26T01:58:59.185492+0000","last_peered":"2024-03-26T01:58:59.185492+0000","last_clean":"2024-03-26T01:58:59.185492+0000","last_became_active":"2024-03-26T01:58:30.134358+0000","last_became_peered":"2024-03-26T01:58:30.134358+0000","last_unstale":"2024-03-26T01:58:59.185492+0000","last_undegraded":"2024-03-26T01:58:59.185492+0000","last_fullsized":"2024-03-26T01:58:59.185492+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:51:21.045204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180437+0000","last_change":"2024-03-26T01:58:32.157227+0000","last_active":"2024-03-26T01:58:59.180437+0000","last_peered":"2024-03-26T01:58:59.180437+0000","last_clean":"2024-03-26T01:58:59.180437+0000","last_became_active":"2024-03-26T01:58:32.155693+0000","last_became_peered":"2024-03-26T01:58:32.155693+0000","last_unstale":"2024-03-26T01:58:59.180437+0000","last_undegraded":"2024-03-26T01:58:59.180437+0000","last_fullsized":"2024-03-26T01:58:59.180437+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:10:38.354365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181521+0000","last_change":"2024-03-26T01:58:36.188246+0000","last_active":"2024-03-26T01:58:59.181521+0000","last_peered":"2024-03-26T01:58:59.181521+0000","last_clean":"2024-03-26T01:58:59.181521+0000","last_became_active":"2024-03-26T01:58:36.186544+0000","last_became_peered":"2024-03-26T01:58:36.186544+0000","last_unstale":"2024-03-26T01:58:59.181521+0000","last_undegraded":"2024-03-26T01:58:59.181521+0000","last_fullsized":"2024-03-26T01:58:59.181521+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:16:56.738069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'8","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:05.186135+0000","last_change":"2024-03-26T01:58:34.170342+0000","last_active":"2024-03-26T01:59:05.186135+0000","last_peered":"2024-03-26T01:59:05.186135+0000","last_clean":"2024-03-26T01:59:05.186135+0000","last_became_active":"2024-03-26T01:58:34.170021+0000","last_became_peered":"2024-03-26T01:58:34.170021+0000","last_unstale":"2024-03-26T01:59:05.186135+0000","last_undegraded":"2024-03-26T01:59:05.186135+0000","last_fullsized":"2024-03-26T01:59:05.186135+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:19:54.773039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184365+0000","last_change":"2024-03-26T01:58:30.134886+0000","last_active":"2024-03-26T01:58:59.184365+0000","last_peered":"2024-03-26T01:58:59.184365+0000","last_clean":"2024-03-26T01:58:59.184365+0000","last_became_active":"2024-03-26T01:58:30.134468+0000","last_became_peered":"2024-03-26T01:58:30.134468+0000","last_unstale":"2024-03-26T01:58:59.184365+0000","last_undegraded":"2024-03-26T01:58:59.184365+0000","last_fullsized":"2024-03-26T01:58:59.184365+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:25:41.189124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181146+0000","last_change":"2024-03-26T01:58:32.159576+0000","last_active":"2024-03-26T01:58:59.181146+0000","last_peered":"2024-03-26T01:58:59.181146+0000","last_clean":"2024-03-26T01:58:59.181146+0000","last_became_active":"2024-03-26T01:58:32.159279+0000","last_became_peered":"2024-03-26T01:58:32.159279+0000","last_unstale":"2024-03-26T01:58:59.181146+0000","last_undegraded":"2024-03-26T01:58:59.181146+0000","last_fullsized":"2024-03-26T01:58:59.181146+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:11:55.406190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181629+0000","last_change":"2024-03-26T01:58:36.186745+0000","last_active":"2024-03-26T01:58:59.181629+0000","last_peered":"2024-03-26T01:58:59.181629+0000","last_clean":"2024-03-26T01:58:59.181629+0000","last_became_active":"2024-03-26T01:58:36.186386+0000","last_became_peered":"2024-03-26T01:58:36.186386+0000","last_unstale":"2024-03-26T01:58:59.181629+0000","last_undegraded":"2024-03-26T01:58:59.181629+0000","last_fullsized":"2024-03-26T01:58:59.181629+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:37:35.511792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182144+0000","last_change":"2024-03-26T01:58:34.175304+0000","last_active":"2024-03-26T01:58:59.182144+0000","last_peered":"2024-03-26T01:58:59.182144+0000","last_clean":"2024-03-26T01:58:59.182144+0000","last_became_active":"2024-03-26T01:58:34.174740+0000","last_became_peered":"2024-03-26T01:58:34.174740+0000","last_unstale":"2024-03-26T01:58:59.182144+0000","last_undegraded":"2024-03-26T01:58:59.182144+0000","last_fullsized":"2024-03-26T01:58:59.182144+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:13:52.155008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180521+0000","last_change":"2024-03-26T01:58:30.139478+0000","last_active":"2024-03-26T01:58:59.180521+0000","last_peered":"2024-03-26T01:58:59.180521+0000","last_clean":"2024-03-26T01:58:59.180521+0000","last_became_active":"2024-03-26T01:58:30.139141+0000","last_became_peered":"2024-03-26T01:58:30.139141+0000","last_unstale":"2024-03-26T01:58:59.180521+0000","last_undegraded":"2024-03-26T01:58:59.180521+0000","last_fullsized":"2024-03-26T01:58:59.180521+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:15:09.336965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"31'4","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180228+0000","last_change":"2024-03-26T01:58:32.161117+0000","last_active":"2024-03-26T01:58:59.180228+0000","last_peered":"2024-03-26T01:58:59.180228+0000","last_clean":"2024-03-26T01:58:59.180228+0000","last_became_active":"2024-03-26T01:58:32.160778+0000","last_became_peered":"2024-03-26T01:58:32.160778+0000","last_unstale":"2024-03-26T01:58:59.180228+0000","last_undegraded":"2024-03-26T01:58:59.180228+0000","last_fullsized":"2024-03-26T01:58:59.180228+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:15:06.507400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181864+0000","last_change":"2024-03-26T01:58:36.190862+0000","last_active":"2024-03-26T01:58:59.181864+0000","last_peered":"2024-03-26T01:58:59.181864+0000","last_clean":"2024-03-26T01:58:59.181864+0000","last_became_active":"2024-03-26T01:58:36.190479+0000","last_became_peered":"2024-03-26T01:58:36.190479+0000","last_unstale":"2024-03-26T01:58:59.181864+0000","last_undegraded":"2024-03-26T01:58:59.181864+0000","last_fullsized":"2024-03-26T01:58:59.181864+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:06:41.524687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182006+0000","last_change":"2024-03-26T01:58:34.173699+0000","last_active":"2024-03-26T01:58:59.182006+0000","last_peered":"2024-03-26T01:58:59.182006+0000","last_clean":"2024-03-26T01:58:59.182006+0000","last_became_active":"2024-03-26T01:58:34.173342+0000","last_became_peered":"2024-03-26T01:58:34.173342+0000","last_unstale":"2024-03-26T01:58:59.182006+0000","last_undegraded":"2024-03-26T01:58:59.182006+0000","last_fullsized":"2024-03-26T01:58:59.182006+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:47:34.991903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183869+0000","last_change":"2024-03-26T01:58:30.138023+0000","last_active":"2024-03-26T01:58:59.183869+0000","last_peered":"2024-03-26T01:58:59.183869+0000","last_clean":"2024-03-26T01:58:59.183869+0000","last_became_active":"2024-03-26T01:58:30.137531+0000","last_became_peered":"2024-03-26T01:58:30.137531+0000","last_unstale":"2024-03-26T01:58:59.183869+0000","last_undegraded":"2024-03-26T01:58:59.183869+0000","last_fullsized":"2024-03-26T01:58:59.183869+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:08:47.173692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"31'17","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183870+0000","last_change":"2024-03-26T01:58:32.156330+0000","last_active":"2024-03-26T01:58:59.183870+0000","last_peered":"2024-03-26T01:58:59.183870+0000","last_clean":"2024-03-26T01:58:59.183870+0000","last_became_active":"2024-03-26T01:58:32.155812+0000","last_became_peered":"2024-03-26T01:58:32.155812+0000","last_unstale":"2024-03-26T01:58:59.183870+0000","last_undegraded":"2024-03-26T01:58:59.183870+0000","last_fullsized":"2024-03-26T01:58:59.183870+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:04:41.451735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184383+0000","last_change":"2024-03-26T01:58:36.190454+0000","last_active":"2024-03-26T01:58:59.184383+0000","last_peered":"2024-03-26T01:58:59.184383+0000","last_clean":"2024-03-26T01:58:59.184383+0000","last_became_active":"2024-03-26T01:58:36.190246+0000","last_became_peered":"2024-03-26T01:58:36.190246+0000","last_unstale":"2024-03-26T01:58:59.184383+0000","last_undegraded":"2024-03-26T01:58:59.184383+0000","last_fullsized":"2024-03-26T01:58:59.184383+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:45:37.113908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"24'2","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.186093+0000","last_change":"2024-03-26T01:58:30.134823+0000","last_active":"2024-03-26T01:58:59.186093+0000","last_peered":"2024-03-26T01:58:59.186093+0000","last_clean":"2024-03-26T01:58:59.186093+0000","last_became_active":"2024-03-26T01:58:30.134496+0000","last_became_peered":"2024-03-26T01:58:30.134496+0000","last_unstale":"2024-03-26T01:58:59.186093+0000","last_undegraded":"2024-03-26T01:58:59.186093+0000","last_fullsized":"2024-03-26T01:58:59.186093+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:58:42.368597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"31'17","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185367+0000","last_change":"2024-03-26T01:58:32.158350+0000","last_active":"2024-03-26T01:58:59.185367+0000","last_peered":"2024-03-26T01:58:59.185367+0000","last_clean":"2024-03-26T01:58:59.185367+0000","last_became_active":"2024-03-26T01:58:32.158043+0000","last_became_peered":"2024-03-26T01:58:32.158043+0000","last_unstale":"2024-03-26T01:58:59.185367+0000","last_undegraded":"2024-03-26T01:58:59.185367+0000","last_fullsized":"2024-03-26T01:58:59.185367+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:09:39.638621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:05.186158+0000","last_change":"2024-03-26T01:58:34.169259+0000","last_active":"2024-03-26T01:59:05.186158+0000","last_peered":"2024-03-26T01:59:05.186158+0000","last_clean":"2024-03-26T01:59:05.186158+0000","last_became_active":"2024-03-26T01:58:34.169027+0000","last_became_peered":"2024-03-26T01:58:34.169027+0000","last_unstale":"2024-03-26T01:59:05.186158+0000","last_undegraded":"2024-03-26T01:59:05.186158+0000","last_fullsized":"2024-03-26T01:59:05.186158+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:43:31.979307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180782+0000","last_change":"2024-03-26T01:58:36.182349+0000","last_active":"2024-03-26T01:58:59.180782+0000","last_peered":"2024-03-26T01:58:59.180782+0000","last_clean":"2024-03-26T01:58:59.180782+0000","last_became_active":"2024-03-26T01:58:36.180850+0000","last_became_peered":"2024-03-26T01:58:36.180850+0000","last_unstale":"2024-03-26T01:58:59.180782+0000","last_undegraded":"2024-03-26T01:58:59.180782+0000","last_fullsized":"2024-03-26T01:58:59.180782+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:18:30.428224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181471+0000","last_change":"2024-03-26T01:58:30.135532+0000","last_active":"2024-03-26T01:58:59.181471+0000","last_peered":"2024-03-26T01:58:59.181471+0000","last_clean":"2024-03-26T01:58:59.181471+0000","last_became_active":"2024-03-26T01:58:30.135209+0000","last_became_peered":"2024-03-26T01:58:30.135209+0000","last_unstale":"2024-03-26T01:58:59.181471+0000","last_undegraded":"2024-03-26T01:58:59.181471+0000","last_fullsized":"2024-03-26T01:58:59.181471+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:00:33.236148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"31'16","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:05.186124+0000","last_change":"2024-03-26T01:58:32.158315+0000","last_active":"2024-03-26T01:59:05.186124+0000","last_peered":"2024-03-26T01:59:05.186124+0000","last_clean":"2024-03-26T01:59:05.186124+0000","last_became_active":"2024-03-26T01:58:32.158073+0000","last_became_peered":"2024-03-26T01:58:32.158073+0000","last_unstale":"2024-03-26T01:59:05.186124+0000","last_undegraded":"2024-03-26T01:59:05.186124+0000","last_fullsized":"2024-03-26T01:59:05.186124+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:41:09.072781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184207+0000","last_change":"2024-03-26T01:58:34.168765+0000","last_active":"2024-03-26T01:58:59.184207+0000","last_peered":"2024-03-26T01:58:59.184207+0000","last_clean":"2024-03-26T01:58:59.184207+0000","last_became_active":"2024-03-26T01:58:34.168408+0000","last_became_peered":"2024-03-26T01:58:34.168408+0000","last_unstale":"2024-03-26T01:58:59.184207+0000","last_undegraded":"2024-03-26T01:58:59.184207+0000","last_fullsized":"2024-03-26T01:58:59.184207+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:45:39.608232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181542+0000","last_change":"2024-03-26T01:58:36.187579+0000","last_active":"2024-03-26T01:58:59.181542+0000","last_peered":"2024-03-26T01:58:59.181542+0000","last_clean":"2024-03-26T01:58:59.181542+0000","last_became_active":"2024-03-26T01:58:36.187213+0000","last_became_peered":"2024-03-26T01:58:36.187213+0000","last_unstale":"2024-03-26T01:58:59.181542+0000","last_undegraded":"2024-03-26T01:58:59.181542+0000","last_fullsized":"2024-03-26T01:58:59.181542+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:32:29.233621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181247+0000","last_change":"2024-03-26T01:58:30.140416+0000","last_active":"2024-03-26T01:58:59.181247+0000","last_peered":"2024-03-26T01:58:59.181247+0000","last_clean":"2024-03-26T01:58:59.181247+0000","last_became_active":"2024-03-26T01:58:30.140021+0000","last_became_peered":"2024-03-26T01:58:30.140021+0000","last_unstale":"2024-03-26T01:58:59.181247+0000","last_undegraded":"2024-03-26T01:58:59.181247+0000","last_fullsized":"2024-03-26T01:58:59.181247+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:27:40.235330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"31'25","reported_seq":71,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181174+0000","last_change":"2024-03-26T01:58:32.159326+0000","last_active":"2024-03-26T01:58:59.181174+0000","last_peered":"2024-03-26T01:58:59.181174+0000","last_clean":"2024-03-26T01:58:59.181174+0000","last_became_active":"2024-03-26T01:58:32.158986+0000","last_became_peered":"2024-03-26T01:58:32.158986+0000","last_unstale":"2024-03-26T01:58:59.181174+0000","last_undegraded":"2024-03-26T01:58:59.181174+0000","last_fullsized":"2024-03-26T01:58:59.181174+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:50:09.892516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181019+0000","last_change":"2024-03-26T01:58:34.172969+0000","last_active":"2024-03-26T01:58:59.181019+0000","last_peered":"2024-03-26T01:58:59.181019+0000","last_clean":"2024-03-26T01:58:59.181019+0000","last_became_active":"2024-03-26T01:58:34.172456+0000","last_became_peered":"2024-03-26T01:58:34.172456+0000","last_unstale":"2024-03-26T01:58:59.181019+0000","last_undegraded":"2024-03-26T01:58:59.181019+0000","last_fullsized":"2024-03-26T01:58:59.181019+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:55:25.288709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182673+0000","last_change":"2024-03-26T01:58:36.191043+0000","last_active":"2024-03-26T01:58:59.182673+0000","last_peered":"2024-03-26T01:58:59.182673+0000","last_clean":"2024-03-26T01:58:59.182673+0000","last_became_active":"2024-03-26T01:58:36.190609+0000","last_became_peered":"2024-03-26T01:58:36.190609+0000","last_unstale":"2024-03-26T01:58:59.182673+0000","last_undegraded":"2024-03-26T01:58:59.182673+0000","last_fullsized":"2024-03-26T01:58:59.182673+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:04:31.833880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"31'30","reported_seq":79,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183486+0000","last_change":"2024-03-26T01:58:32.158264+0000","last_active":"2024-03-26T01:58:59.183486+0000","last_peered":"2024-03-26T01:58:59.183486+0000","last_clean":"2024-03-26T01:58:59.183486+0000","last_became_active":"2024-03-26T01:58:32.157952+0000","last_became_peered":"2024-03-26T01:58:32.157952+0000","last_unstale":"2024-03-26T01:58:59.183486+0000","last_undegraded":"2024-03-26T01:58:59.183486+0000","last_fullsized":"2024-03-26T01:58:59.183486+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:19:27.115554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179608+0000","last_change":"2024-03-26T01:58:30.136558+0000","last_active":"2024-03-26T01:58:59.179608+0000","last_peered":"2024-03-26T01:58:59.179608+0000","last_clean":"2024-03-26T01:58:59.179608+0000","last_became_active":"2024-03-26T01:58:30.135134+0000","last_became_peered":"2024-03-26T01:58:30.135134+0000","last_unstale":"2024-03-26T01:58:59.179608+0000","last_undegraded":"2024-03-26T01:58:59.179608+0000","last_fullsized":"2024-03-26T01:58:59.179608+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:49:56.542251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182727+0000","last_change":"2024-03-26T01:58:34.174621+0000","last_active":"2024-03-26T01:58:59.182727+0000","last_peered":"2024-03-26T01:58:59.182727+0000","last_clean":"2024-03-26T01:58:59.182727+0000","last_became_active":"2024-03-26T01:58:34.173408+0000","last_became_peered":"2024-03-26T01:58:34.173408+0000","last_unstale":"2024-03-26T01:58:59.182727+0000","last_undegraded":"2024-03-26T01:58:59.182727+0000","last_fullsized":"2024-03-26T01:58:59.182727+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:27:47.938754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184612+0000","last_change":"2024-03-26T01:58:36.182933+0000","last_active":"2024-03-26T01:58:59.184612+0000","last_peered":"2024-03-26T01:58:59.184612+0000","last_clean":"2024-03-26T01:58:59.184612+0000","last_became_active":"2024-03-26T01:58:36.182576+0000","last_became_peered":"2024-03-26T01:58:36.182576+0000","last_unstale":"2024-03-26T01:58:59.184612+0000","last_undegraded":"2024-03-26T01:58:59.184612+0000","last_fullsized":"2024-03-26T01:58:59.184612+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:25:11.397494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"31'16","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180340+0000","last_change":"2024-03-26T01:58:32.156762+0000","last_active":"2024-03-26T01:58:59.180340+0000","last_peered":"2024-03-26T01:58:59.180340+0000","last_clean":"2024-03-26T01:58:59.180340+0000","last_became_active":"2024-03-26T01:58:32.155408+0000","last_became_peered":"2024-03-26T01:58:32.155408+0000","last_unstale":"2024-03-26T01:58:59.180340+0000","last_undegraded":"2024-03-26T01:58:59.180340+0000","last_fullsized":"2024-03-26T01:58:59.180340+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:47:27.517461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181334+0000","last_change":"2024-03-26T01:58:30.135224+0000","last_active":"2024-03-26T01:58:59.181334+0000","last_peered":"2024-03-26T01:58:59.181334+0000","last_clean":"2024-03-26T01:58:59.181334+0000","last_became_active":"2024-03-26T01:58:30.134183+0000","last_became_peered":"2024-03-26T01:58:30.134183+0000","last_unstale":"2024-03-26T01:58:59.181334+0000","last_undegraded":"2024-03-26T01:58:59.181334+0000","last_fullsized":"2024-03-26T01:58:59.181334+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:46:24.952993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"32'208","reported_seq":314,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:01.277658+0000","last_change":"2024-03-26T01:58:20.077262+0000","last_active":"2024-03-26T01:59:01.277658+0000","last_peered":"2024-03-26T01:59:01.277658+0000","last_clean":"2024-03-26T01:59:01.277658+0000","last_became_active":"2024-03-26T01:58:20.076923+0000","last_became_peered":"2024-03-26T01:58:20.076923+0000","last_unstale":"2024-03-26T01:59:01.277658+0000","last_undegraded":"2024-03-26T01:59:01.277658+0000","last_fullsized":"2024-03-26T01:59:01.277658+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:19.055798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:19.055798+0000","last_clean_scrub_stamp":"2024-03-26T01:58:19.055798+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:52:02.430554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181954+0000","last_change":"2024-03-26T01:58:34.167011+0000","last_active":"2024-03-26T01:58:59.181954+0000","last_peered":"2024-03-26T01:58:59.181954+0000","last_clean":"2024-03-26T01:58:59.181954+0000","last_became_active":"2024-03-26T01:58:34.166630+0000","last_became_peered":"2024-03-26T01:58:34.166630+0000","last_unstale":"2024-03-26T01:58:59.181954+0000","last_undegraded":"2024-03-26T01:58:59.181954+0000","last_fullsized":"2024-03-26T01:58:59.181954+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:40:17.454107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181747+0000","last_change":"2024-03-26T01:58:36.187993+0000","last_active":"2024-03-26T01:58:59.181747+0000","last_peered":"2024-03-26T01:58:59.181747+0000","last_clean":"2024-03-26T01:58:59.181747+0000","last_became_active":"2024-03-26T01:58:36.186411+0000","last_became_peered":"2024-03-26T01:58:36.186411+0000","last_unstale":"2024-03-26T01:58:59.181747+0000","last_undegraded":"2024-03-26T01:58:59.181747+0000","last_fullsized":"2024-03-26T01:58:59.181747+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:40:49.335258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"31'20","reported_seq":56,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179835+0000","last_change":"2024-03-26T01:58:32.156747+0000","last_active":"2024-03-26T01:58:59.179835+0000","last_peered":"2024-03-26T01:58:59.179835+0000","last_clean":"2024-03-26T01:58:59.179835+0000","last_became_active":"2024-03-26T01:58:32.155645+0000","last_became_peered":"2024-03-26T01:58:32.155645+0000","last_unstale":"2024-03-26T01:58:59.179835+0000","last_undegraded":"2024-03-26T01:58:59.179835+0000","last_fullsized":"2024-03-26T01:58:59.179835+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:07:21.232257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180114+0000","last_change":"2024-03-26T01:58:30.135446+0000","last_active":"2024-03-26T01:58:59.180114+0000","last_peered":"2024-03-26T01:58:59.180114+0000","last_clean":"2024-03-26T01:58:59.180114+0000","last_became_active":"2024-03-26T01:58:30.134026+0000","last_became_peered":"2024-03-26T01:58:30.134026+0000","last_unstale":"2024-03-26T01:58:59.180114+0000","last_undegraded":"2024-03-26T01:58:59.180114+0000","last_fullsized":"2024-03-26T01:58:59.180114+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:31:00.260417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184415+0000","last_change":"2024-03-26T01:58:34.171893+0000","last_active":"2024-03-26T01:58:59.184415+0000","last_peered":"2024-03-26T01:58:59.184415+0000","last_clean":"2024-03-26T01:58:59.184415+0000","last_became_active":"2024-03-26T01:58:34.171607+0000","last_became_peered":"2024-03-26T01:58:34.171607+0000","last_unstale":"2024-03-26T01:58:59.184415+0000","last_undegraded":"2024-03-26T01:58:59.184415+0000","last_fullsized":"2024-03-26T01:58:59.184415+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:36:47.600932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181055+0000","last_change":"2024-03-26T01:58:36.186337+0000","last_active":"2024-03-26T01:58:59.181055+0000","last_peered":"2024-03-26T01:58:59.181055+0000","last_clean":"2024-03-26T01:58:59.181055+0000","last_became_active":"2024-03-26T01:58:36.185156+0000","last_became_peered":"2024-03-26T01:58:36.185156+0000","last_unstale":"2024-03-26T01:58:59.181055+0000","last_undegraded":"2024-03-26T01:58:59.181055+0000","last_fullsized":"2024-03-26T01:58:59.181055+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:39:30.381455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.186270+0000","last_change":"2024-03-26T01:58:30.139116+0000","last_active":"2024-03-26T01:58:59.186270+0000","last_peered":"2024-03-26T01:58:59.186270+0000","last_clean":"2024-03-26T01:58:59.186270+0000","last_became_active":"2024-03-26T01:58:30.137541+0000","last_became_peered":"2024-03-26T01:58:30.137541+0000","last_unstale":"2024-03-26T01:58:59.186270+0000","last_undegraded":"2024-03-26T01:58:59.186270+0000","last_fullsized":"2024-03-26T01:58:59.186270+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:02:15.121649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'34","reported_seq":92,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:05.185811+0000","last_change":"2024-03-26T01:58:32.154799+0000","last_active":"2024-03-26T01:59:05.185811+0000","last_peered":"2024-03-26T01:59:05.185811+0000","last_clean":"2024-03-26T01:59:05.185811+0000","last_became_active":"2024-03-26T01:58:32.154338+0000","last_became_peered":"2024-03-26T01:58:32.154338+0000","last_unstale":"2024-03-26T01:59:05.185811+0000","last_undegraded":"2024-03-26T01:59:05.185811+0000","last_fullsized":"2024-03-26T01:59:05.185811+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:57:29.818605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:05.186086+0000","last_change":"2024-03-26T01:58:34.168792+0000","last_active":"2024-03-26T01:59:05.186086+0000","last_peered":"2024-03-26T01:59:05.186086+0000","last_clean":"2024-03-26T01:59:05.186086+0000","last_became_active":"2024-03-26T01:58:34.168445+0000","last_became_peered":"2024-03-26T01:58:34.168445+0000","last_unstale":"2024-03-26T01:59:05.186086+0000","last_undegraded":"2024-03-26T01:59:05.186086+0000","last_fullsized":"2024-03-26T01:59:05.186086+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:47:49.238420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179928+0000","last_change":"2024-03-26T01:58:36.188257+0000","last_active":"2024-03-26T01:58:59.179928+0000","last_peered":"2024-03-26T01:58:59.179928+0000","last_clean":"2024-03-26T01:58:59.179928+0000","last_became_active":"2024-03-26T01:58:36.186442+0000","last_became_peered":"2024-03-26T01:58:36.186442+0000","last_unstale":"2024-03-26T01:58:59.179928+0000","last_undegraded":"2024-03-26T01:58:59.179928+0000","last_fullsized":"2024-03-26T01:58:59.179928+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:17:03.649301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182228+0000","last_change":"2024-03-26T01:58:30.136657+0000","last_active":"2024-03-26T01:58:59.182228+0000","last_peered":"2024-03-26T01:58:59.182228+0000","last_clean":"2024-03-26T01:58:59.182228+0000","last_became_active":"2024-03-26T01:58:30.135563+0000","last_became_peered":"2024-03-26T01:58:30.135563+0000","last_unstale":"2024-03-26T01:58:59.182228+0000","last_undegraded":"2024-03-26T01:58:59.182228+0000","last_fullsized":"2024-03-26T01:58:59.182228+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:30:56.449784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"31'19","reported_seq":62,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183016+0000","last_change":"2024-03-26T01:58:32.157561+0000","last_active":"2024-03-26T01:58:59.183016+0000","last_peered":"2024-03-26T01:58:59.183016+0000","last_clean":"2024-03-26T01:58:59.183016+0000","last_became_active":"2024-03-26T01:58:32.156539+0000","last_became_peered":"2024-03-26T01:58:32.156539+0000","last_unstale":"2024-03-26T01:58:59.183016+0000","last_undegraded":"2024-03-26T01:58:59.183016+0000","last_fullsized":"2024-03-26T01:58:59.183016+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:52:01.988500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:05.185791+0000","last_change":"2024-03-26T01:58:34.175092+0000","last_active":"2024-03-26T01:59:05.185791+0000","last_peered":"2024-03-26T01:59:05.185791+0000","last_clean":"2024-03-26T01:59:05.185791+0000","last_became_active":"2024-03-26T01:58:34.174745+0000","last_became_peered":"2024-03-26T01:58:34.174745+0000","last_unstale":"2024-03-26T01:59:05.185791+0000","last_undegraded":"2024-03-26T01:59:05.185791+0000","last_fullsized":"2024-03-26T01:59:05.185791+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:39:28.143023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184283+0000","last_change":"2024-03-26T01:58:36.189161+0000","last_active":"2024-03-26T01:58:59.184283+0000","last_peered":"2024-03-26T01:58:59.184283+0000","last_clean":"2024-03-26T01:58:59.184283+0000","last_became_active":"2024-03-26T01:58:36.188783+0000","last_became_peered":"2024-03-26T01:58:36.188783+0000","last_unstale":"2024-03-26T01:58:59.184283+0000","last_undegraded":"2024-03-26T01:58:59.184283+0000","last_fullsized":"2024-03-26T01:58:59.184283+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:39:11.216002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183015+0000","last_change":"2024-03-26T01:58:30.140194+0000","last_active":"2024-03-26T01:58:59.183015+0000","last_peered":"2024-03-26T01:58:59.183015+0000","last_clean":"2024-03-26T01:58:59.183015+0000","last_became_active":"2024-03-26T01:58:30.139697+0000","last_became_peered":"2024-03-26T01:58:30.139697+0000","last_unstale":"2024-03-26T01:58:59.183015+0000","last_undegraded":"2024-03-26T01:58:59.183015+0000","last_fullsized":"2024-03-26T01:58:59.183015+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:41:19.704783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"31'12","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182348+0000","last_change":"2024-03-26T01:58:32.157225+0000","last_active":"2024-03-26T01:58:59.182348+0000","last_peered":"2024-03-26T01:58:59.182348+0000","last_clean":"2024-03-26T01:58:59.182348+0000","last_became_active":"2024-03-26T01:58:32.156161+0000","last_became_peered":"2024-03-26T01:58:32.156161+0000","last_unstale":"2024-03-26T01:58:59.182348+0000","last_undegraded":"2024-03-26T01:58:59.182348+0000","last_fullsized":"2024-03-26T01:58:59.182348+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:28:30.498738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183686+0000","last_change":"2024-03-26T01:58:34.168744+0000","last_active":"2024-03-26T01:58:59.183686+0000","last_peered":"2024-03-26T01:58:59.183686+0000","last_clean":"2024-03-26T01:58:59.183686+0000","last_became_active":"2024-03-26T01:58:34.168398+0000","last_became_peered":"2024-03-26T01:58:34.168398+0000","last_unstale":"2024-03-26T01:58:59.183686+0000","last_undegraded":"2024-03-26T01:58:59.183686+0000","last_fullsized":"2024-03-26T01:58:59.183686+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:49:37.806127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181911+0000","last_change":"2024-03-26T01:58:36.188194+0000","last_active":"2024-03-26T01:58:59.181911+0000","last_peered":"2024-03-26T01:58:59.181911+0000","last_clean":"2024-03-26T01:58:59.181911+0000","last_became_active":"2024-03-26T01:58:36.186464+0000","last_became_peered":"2024-03-26T01:58:36.186464+0000","last_unstale":"2024-03-26T01:58:59.181911+0000","last_undegraded":"2024-03-26T01:58:59.181911+0000","last_fullsized":"2024-03-26T01:58:59.181911+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:52:58.024240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:05.186015+0000","last_change":"2024-03-26T01:58:34.171383+0000","last_active":"2024-03-26T01:59:05.186015+0000","last_peered":"2024-03-26T01:59:05.186015+0000","last_clean":"2024-03-26T01:59:05.186015+0000","last_became_active":"2024-03-26T01:58:34.170560+0000","last_became_peered":"2024-03-26T01:58:34.170560+0000","last_unstale":"2024-03-26T01:59:05.186015+0000","last_undegraded":"2024-03-26T01:59:05.186015+0000","last_fullsized":"2024-03-26T01:59:05.186015+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:15:55.206007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179953+0000","last_change":"2024-03-26T01:58:30.135491+0000","last_active":"2024-03-26T01:58:59.179953+0000","last_peered":"2024-03-26T01:58:59.179953+0000","last_clean":"2024-03-26T01:58:59.179953+0000","last_became_active":"2024-03-26T01:58:30.134958+0000","last_became_peered":"2024-03-26T01:58:30.134958+0000","last_unstale":"2024-03-26T01:58:59.179953+0000","last_undegraded":"2024-03-26T01:58:59.179953+0000","last_fullsized":"2024-03-26T01:58:59.179953+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:03:32.726569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"31'24","reported_seq":70,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184139+0000","last_change":"2024-03-26T01:58:32.157117+0000","last_active":"2024-03-26T01:58:59.184139+0000","last_peered":"2024-03-26T01:58:59.184139+0000","last_clean":"2024-03-26T01:58:59.184139+0000","last_became_active":"2024-03-26T01:58:32.156788+0000","last_became_peered":"2024-03-26T01:58:32.156788+0000","last_unstale":"2024-03-26T01:58:59.184139+0000","last_undegraded":"2024-03-26T01:58:59.184139+0000","last_fullsized":"2024-03-26T01:58:59.184139+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:17:29.181591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184537+0000","last_change":"2024-03-26T01:58:36.183880+0000","last_active":"2024-03-26T01:58:59.184537+0000","last_peered":"2024-03-26T01:58:59.184537+0000","last_clean":"2024-03-26T01:58:59.184537+0000","last_became_active":"2024-03-26T01:58:36.183518+0000","last_became_peered":"2024-03-26T01:58:36.183518+0000","last_unstale":"2024-03-26T01:58:59.184537+0000","last_undegraded":"2024-03-26T01:58:59.184537+0000","last_fullsized":"2024-03-26T01:58:59.184537+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:43:45.259076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182232+0000","last_change":"2024-03-26T01:58:34.174097+0000","last_active":"2024-03-26T01:58:59.182232+0000","last_peered":"2024-03-26T01:58:59.182232+0000","last_clean":"2024-03-26T01:58:59.182232+0000","last_became_active":"2024-03-26T01:58:34.173258+0000","last_became_peered":"2024-03-26T01:58:34.173258+0000","last_unstale":"2024-03-26T01:58:59.182232+0000","last_undegraded":"2024-03-26T01:58:59.182232+0000","last_fullsized":"2024-03-26T01:58:59.182232+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:24:35.674377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179664+0000","last_change":"2024-03-26T01:58:30.135467+0000","last_active":"2024-03-26T01:58:59.179664+0000","last_peered":"2024-03-26T01:58:59.179664+0000","last_clean":"2024-03-26T01:58:59.179664+0000","last_became_active":"2024-03-26T01:58:30.135083+0000","last_became_peered":"2024-03-26T01:58:30.135083+0000","last_unstale":"2024-03-26T01:58:59.179664+0000","last_undegraded":"2024-03-26T01:58:59.179664+0000","last_fullsized":"2024-03-26T01:58:59.179664+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:45:14.791345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'28","reported_seq":73,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183860+0000","last_change":"2024-03-26T01:58:32.157107+0000","last_active":"2024-03-26T01:58:59.183860+0000","last_peered":"2024-03-26T01:58:59.183860+0000","last_clean":"2024-03-26T01:58:59.183860+0000","last_became_active":"2024-03-26T01:58:32.156812+0000","last_became_peered":"2024-03-26T01:58:32.156812+0000","last_unstale":"2024-03-26T01:58:59.183860+0000","last_undegraded":"2024-03-26T01:58:59.183860+0000","last_fullsized":"2024-03-26T01:58:59.183860+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:01:12.654416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185909+0000","last_change":"2024-03-26T01:58:36.183732+0000","last_active":"2024-03-26T01:58:59.185909+0000","last_peered":"2024-03-26T01:58:59.185909+0000","last_clean":"2024-03-26T01:58:59.185909+0000","last_became_active":"2024-03-26T01:58:36.183404+0000","last_became_peered":"2024-03-26T01:58:36.183404+0000","last_unstale":"2024-03-26T01:58:59.185909+0000","last_undegraded":"2024-03-26T01:58:59.185909+0000","last_fullsized":"2024-03-26T01:58:59.185909+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:54:49.610997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184825+0000","last_change":"2024-03-26T01:58:30.134700+0000","last_active":"2024-03-26T01:58:59.184825+0000","last_peered":"2024-03-26T01:58:59.184825+0000","last_clean":"2024-03-26T01:58:59.184825+0000","last_became_active":"2024-03-26T01:58:30.134336+0000","last_became_peered":"2024-03-26T01:58:30.134336+0000","last_unstale":"2024-03-26T01:58:59.184825+0000","last_undegraded":"2024-03-26T01:58:59.184825+0000","last_fullsized":"2024-03-26T01:58:59.184825+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:49:13.401189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"31'15","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181251+0000","last_change":"2024-03-26T01:58:32.156397+0000","last_active":"2024-03-26T01:58:59.181251+0000","last_peered":"2024-03-26T01:58:59.181251+0000","last_clean":"2024-03-26T01:58:59.181251+0000","last_became_active":"2024-03-26T01:58:32.155515+0000","last_became_peered":"2024-03-26T01:58:32.155515+0000","last_unstale":"2024-03-26T01:58:59.181251+0000","last_undegraded":"2024-03-26T01:58:59.181251+0000","last_fullsized":"2024-03-26T01:58:59.181251+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:30:51.951393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181761+0000","last_change":"2024-03-26T01:58:34.170142+0000","last_active":"2024-03-26T01:58:59.181761+0000","last_peered":"2024-03-26T01:58:59.181761+0000","last_clean":"2024-03-26T01:58:59.181761+0000","last_became_active":"2024-03-26T01:58:34.169668+0000","last_became_peered":"2024-03-26T01:58:34.169668+0000","last_unstale":"2024-03-26T01:58:59.181761+0000","last_undegraded":"2024-03-26T01:58:59.181761+0000","last_fullsized":"2024-03-26T01:58:59.181761+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:58:32.525102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180922+0000","last_change":"2024-03-26T01:58:36.183224+0000","last_active":"2024-03-26T01:58:59.180922+0000","last_peered":"2024-03-26T01:58:59.180922+0000","last_clean":"2024-03-26T01:58:59.180922+0000","last_became_active":"2024-03-26T01:58:36.182210+0000","last_became_peered":"2024-03-26T01:58:36.182210+0000","last_unstale":"2024-03-26T01:58:59.180922+0000","last_undegraded":"2024-03-26T01:58:59.180922+0000","last_fullsized":"2024-03-26T01:58:59.180922+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:40:42.882517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:05.186124+0000","last_change":"2024-03-26T01:58:34.167283+0000","last_active":"2024-03-26T01:59:05.186124+0000","last_peered":"2024-03-26T01:59:05.186124+0000","last_clean":"2024-03-26T01:59:05.186124+0000","last_became_active":"2024-03-26T01:58:34.166709+0000","last_became_peered":"2024-03-26T01:58:34.166709+0000","last_unstale":"2024-03-26T01:59:05.186124+0000","last_undegraded":"2024-03-26T01:59:05.186124+0000","last_fullsized":"2024-03-26T01:59:05.186124+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:03:24.270177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185647+0000","last_change":"2024-03-26T01:58:30.136006+0000","last_active":"2024-03-26T01:58:59.185647+0000","last_peered":"2024-03-26T01:58:59.185647+0000","last_clean":"2024-03-26T01:58:59.185647+0000","last_became_active":"2024-03-26T01:58:30.135630+0000","last_became_peered":"2024-03-26T01:58:30.135630+0000","last_unstale":"2024-03-26T01:58:59.185647+0000","last_undegraded":"2024-03-26T01:58:59.185647+0000","last_fullsized":"2024-03-26T01:58:59.185647+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:28:02.175466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"31'25","reported_seq":66,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181034+0000","last_change":"2024-03-26T01:58:32.160879+0000","last_active":"2024-03-26T01:58:59.181034+0000","last_peered":"2024-03-26T01:58:59.181034+0000","last_clean":"2024-03-26T01:58:59.181034+0000","last_became_active":"2024-03-26T01:58:32.160577+0000","last_became_peered":"2024-03-26T01:58:32.160577+0000","last_unstale":"2024-03-26T01:58:59.181034+0000","last_undegraded":"2024-03-26T01:58:59.181034+0000","last_fullsized":"2024-03-26T01:58:59.181034+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:46:59.282899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185138+0000","last_change":"2024-03-26T01:58:36.183872+0000","last_active":"2024-03-26T01:58:59.185138+0000","last_peered":"2024-03-26T01:58:59.185138+0000","last_clean":"2024-03-26T01:58:59.185138+0000","last_became_active":"2024-03-26T01:58:36.183540+0000","last_became_peered":"2024-03-26T01:58:36.183540+0000","last_unstale":"2024-03-26T01:58:59.185138+0000","last_undegraded":"2024-03-26T01:58:59.185138+0000","last_fullsized":"2024-03-26T01:58:59.185138+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:00:46.373628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185409+0000","last_change":"2024-03-26T01:58:30.137193+0000","last_active":"2024-03-26T01:58:59.185409+0000","last_peered":"2024-03-26T01:58:59.185409+0000","last_clean":"2024-03-26T01:58:59.185409+0000","last_became_active":"2024-03-26T01:58:30.135632+0000","last_became_peered":"2024-03-26T01:58:30.135632+0000","last_unstale":"2024-03-26T01:58:59.185409+0000","last_undegraded":"2024-03-26T01:58:59.185409+0000","last_fullsized":"2024-03-26T01:58:59.185409+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:41:10.153458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'17","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182950+0000","last_change":"2024-03-26T01:58:32.157715+0000","last_active":"2024-03-26T01:58:59.182950+0000","last_peered":"2024-03-26T01:58:59.182950+0000","last_clean":"2024-03-26T01:58:59.182950+0000","last_became_active":"2024-03-26T01:58:32.157410+0000","last_became_peered":"2024-03-26T01:58:32.157410+0000","last_unstale":"2024-03-26T01:58:59.182950+0000","last_undegraded":"2024-03-26T01:58:59.182950+0000","last_fullsized":"2024-03-26T01:58:59.182950+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:07:07.497045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180964+0000","last_change":"2024-03-26T01:58:34.165811+0000","last_active":"2024-03-26T01:58:59.180964+0000","last_peered":"2024-03-26T01:58:59.180964+0000","last_clean":"2024-03-26T01:58:59.180964+0000","last_became_active":"2024-03-26T01:58:34.165511+0000","last_became_peered":"2024-03-26T01:58:34.165511+0000","last_unstale":"2024-03-26T01:58:59.180964+0000","last_undegraded":"2024-03-26T01:58:59.180964+0000","last_fullsized":"2024-03-26T01:58:59.180964+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:13:25.736549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179952+0000","last_change":"2024-03-26T01:58:36.183039+0000","last_active":"2024-03-26T01:58:59.179952+0000","last_peered":"2024-03-26T01:58:59.179952+0000","last_clean":"2024-03-26T01:58:59.179952+0000","last_became_active":"2024-03-26T01:58:36.182662+0000","last_became_peered":"2024-03-26T01:58:36.182662+0000","last_unstale":"2024-03-26T01:58:59.179952+0000","last_undegraded":"2024-03-26T01:58:59.179952+0000","last_fullsized":"2024-03-26T01:58:59.179952+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:00:42.354013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.186104+0000","last_change":"2024-03-26T01:58:30.139735+0000","last_active":"2024-03-26T01:58:59.186104+0000","last_peered":"2024-03-26T01:58:59.186104+0000","last_clean":"2024-03-26T01:58:59.186104+0000","last_became_active":"2024-03-26T01:58:30.138545+0000","last_became_peered":"2024-03-26T01:58:30.138545+0000","last_unstale":"2024-03-26T01:58:59.186104+0000","last_undegraded":"2024-03-26T01:58:59.186104+0000","last_fullsized":"2024-03-26T01:58:59.186104+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:41:32.908074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182862+0000","last_change":"2024-03-26T01:58:32.154564+0000","last_active":"2024-03-26T01:58:59.182862+0000","last_peered":"2024-03-26T01:58:59.182862+0000","last_clean":"2024-03-26T01:58:59.182862+0000","last_became_active":"2024-03-26T01:58:32.154194+0000","last_became_peered":"2024-03-26T01:58:32.154194+0000","last_unstale":"2024-03-26T01:58:59.182862+0000","last_undegraded":"2024-03-26T01:58:59.182862+0000","last_fullsized":"2024-03-26T01:58:59.182862+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:14:57.940064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180680+0000","last_change":"2024-03-26T01:58:34.171756+0000","last_active":"2024-03-26T01:58:59.180680+0000","last_peered":"2024-03-26T01:58:59.180680+0000","last_clean":"2024-03-26T01:58:59.180680+0000","last_became_active":"2024-03-26T01:58:34.171327+0000","last_became_peered":"2024-03-26T01:58:34.171327+0000","last_unstale":"2024-03-26T01:58:59.180680+0000","last_undegraded":"2024-03-26T01:58:59.180680+0000","last_fullsized":"2024-03-26T01:58:59.180680+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:58:38.598302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180170+0000","last_change":"2024-03-26T01:58:36.188667+0000","last_active":"2024-03-26T01:58:59.180170+0000","last_peered":"2024-03-26T01:58:59.180170+0000","last_clean":"2024-03-26T01:58:59.180170+0000","last_became_active":"2024-03-26T01:58:36.187974+0000","last_became_peered":"2024-03-26T01:58:36.187974+0000","last_unstale":"2024-03-26T01:58:59.180170+0000","last_undegraded":"2024-03-26T01:58:59.180170+0000","last_fullsized":"2024-03-26T01:58:59.180170+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:58:10.007013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182823+0000","last_change":"2024-03-26T01:58:30.137553+0000","last_active":"2024-03-26T01:58:59.182823+0000","last_peered":"2024-03-26T01:58:59.182823+0000","last_clean":"2024-03-26T01:58:59.182823+0000","last_became_active":"2024-03-26T01:58:30.137125+0000","last_became_peered":"2024-03-26T01:58:30.137125+0000","last_unstale":"2024-03-26T01:58:59.182823+0000","last_undegraded":"2024-03-26T01:58:59.182823+0000","last_fullsized":"2024-03-26T01:58:59.182823+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:21:29.748266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183184+0000","last_change":"2024-03-26T01:58:32.159537+0000","last_active":"2024-03-26T01:58:59.183184+0000","last_peered":"2024-03-26T01:58:59.183184+0000","last_clean":"2024-03-26T01:58:59.183184+0000","last_became_active":"2024-03-26T01:58:32.159254+0000","last_became_peered":"2024-03-26T01:58:32.159254+0000","last_unstale":"2024-03-26T01:58:59.183184+0000","last_undegraded":"2024-03-26T01:58:59.183184+0000","last_fullsized":"2024-03-26T01:58:59.183184+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:20:35.912710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184124+0000","last_change":"2024-03-26T01:58:34.171491+0000","last_active":"2024-03-26T01:58:59.184124+0000","last_peered":"2024-03-26T01:58:59.184124+0000","last_clean":"2024-03-26T01:58:59.184124+0000","last_became_active":"2024-03-26T01:58:34.171153+0000","last_became_peered":"2024-03-26T01:58:34.171153+0000","last_unstale":"2024-03-26T01:58:59.184124+0000","last_undegraded":"2024-03-26T01:58:59.184124+0000","last_fullsized":"2024-03-26T01:58:59.184124+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:09:37.497089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":590,"ondisk_log_size":590,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411340,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738390,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T01:59:13.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T01:59:13.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:12 smithi003 ceph-mon[25810]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T01:59:13.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:12 smithi003 ceph-mon[25810]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:59:13.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:12 smithi066 ceph-mon[30259]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T01:59:13.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:12 smithi066 ceph-mon[30259]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:59:13.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:12 smithi055 ceph-mon[30619]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T01:59:13.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:12 smithi055 ceph-mon[30619]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:59:13.284 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:59:14.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:14 smithi003 ceph-mon[25810]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T01:59:14.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:14 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:59:15.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:14 smithi066 ceph-mon[30259]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T01:59:15.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:14 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:59:15.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:14 smithi055 ceph-mon[30619]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T01:59:15.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:14 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:59:17.095 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:16 smithi003 ceph-mon[25810]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:17.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:16 smithi066 ceph-mon[30259]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:17.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:16 smithi055 ceph-mon[30619]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:17.873 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:17 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: ::ffff:172.21.15.55 - - [26/Mar/2024:01:59:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T01:59:18.243 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:59:18.246 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-26T01:59:18.528 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 01:59:18 smithi003 systemd[1]: Starting Ceph alertmanager.a for b95eae18-eb13-11ee-95d0-87774f69a715... 2024-03-26T01:59:18.795 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-03-26T01:59:17.194831+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":1262,"num_read_kb":1077,"num_write":889,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":870,"ondisk_log_size":870,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96636,"kb_used_data":3108,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126532,"statfs":{"total":287972524032,"available":287873568768,"internally_reserved":0,"allocated":3182592,"data_stored":2109504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002877"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180531+0000","last_change":"2024-03-26T01:58:36.186063+0000","last_active":"2024-03-26T01:58:59.180531+0000","last_peered":"2024-03-26T01:58:59.180531+0000","last_clean":"2024-03-26T01:58:59.180531+0000","last_became_active":"2024-03-26T01:58:36.184564+0000","last_became_peered":"2024-03-26T01:58:36.184564+0000","last_unstale":"2024-03-26T01:58:59.180531+0000","last_undegraded":"2024-03-26T01:58:59.180531+0000","last_fullsized":"2024-03-26T01:58:59.180531+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:58:40.986608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183574+0000","last_change":"2024-03-26T01:58:34.169468+0000","last_active":"2024-03-26T01:58:59.183574+0000","last_peered":"2024-03-26T01:58:59.183574+0000","last_clean":"2024-03-26T01:58:59.183574+0000","last_became_active":"2024-03-26T01:58:34.169220+0000","last_became_peered":"2024-03-26T01:58:34.169220+0000","last_unstale":"2024-03-26T01:58:59.183574+0000","last_undegraded":"2024-03-26T01:58:59.183574+0000","last_fullsized":"2024-03-26T01:58:59.183574+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:21:02.472833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182555+0000","last_change":"2024-03-26T01:58:30.136433+0000","last_active":"2024-03-26T01:58:59.182555+0000","last_peered":"2024-03-26T01:58:59.182555+0000","last_clean":"2024-03-26T01:58:59.182555+0000","last_became_active":"2024-03-26T01:58:30.136121+0000","last_became_peered":"2024-03-26T01:58:30.136121+0000","last_unstale":"2024-03-26T01:58:59.182555+0000","last_undegraded":"2024-03-26T01:58:59.182555+0000","last_fullsized":"2024-03-26T01:58:59.182555+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:13:26.702227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"31'16","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182913+0000","last_change":"2024-03-26T01:58:32.155738+0000","last_active":"2024-03-26T01:58:59.182913+0000","last_peered":"2024-03-26T01:58:59.182913+0000","last_clean":"2024-03-26T01:58:59.182913+0000","last_became_active":"2024-03-26T01:58:32.154739+0000","last_became_peered":"2024-03-26T01:58:32.154739+0000","last_unstale":"2024-03-26T01:58:59.182913+0000","last_undegraded":"2024-03-26T01:58:59.182913+0000","last_fullsized":"2024-03-26T01:58:59.182913+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:15:16.182003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179615+0000","last_change":"2024-03-26T01:58:36.182482+0000","last_active":"2024-03-26T01:58:59.179615+0000","last_peered":"2024-03-26T01:58:59.179615+0000","last_clean":"2024-03-26T01:58:59.179615+0000","last_became_active":"2024-03-26T01:58:36.182070+0000","last_became_peered":"2024-03-26T01:58:36.182070+0000","last_unstale":"2024-03-26T01:58:59.179615+0000","last_undegraded":"2024-03-26T01:58:59.179615+0000","last_fullsized":"2024-03-26T01:58:59.179615+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:22:30.388786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180379+0000","last_change":"2024-03-26T01:58:34.167613+0000","last_active":"2024-03-26T01:58:59.180379+0000","last_peered":"2024-03-26T01:58:59.180379+0000","last_clean":"2024-03-26T01:58:59.180379+0000","last_became_active":"2024-03-26T01:58:34.166603+0000","last_became_peered":"2024-03-26T01:58:34.166603+0000","last_unstale":"2024-03-26T01:58:59.180379+0000","last_undegraded":"2024-03-26T01:58:59.180379+0000","last_fullsized":"2024-03-26T01:58:59.180379+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:40:17.393707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182555+0000","last_change":"2024-03-26T01:58:30.137133+0000","last_active":"2024-03-26T01:58:59.182555+0000","last_peered":"2024-03-26T01:58:59.182555+0000","last_clean":"2024-03-26T01:58:59.182555+0000","last_became_active":"2024-03-26T01:58:30.136181+0000","last_became_peered":"2024-03-26T01:58:30.136181+0000","last_unstale":"2024-03-26T01:58:59.182555+0000","last_undegraded":"2024-03-26T01:58:59.182555+0000","last_fullsized":"2024-03-26T01:58:59.182555+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:27:18.062383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'17","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182336+0000","last_change":"2024-03-26T01:58:32.157641+0000","last_active":"2024-03-26T01:58:59.182336+0000","last_peered":"2024-03-26T01:58:59.182336+0000","last_clean":"2024-03-26T01:58:59.182336+0000","last_became_active":"2024-03-26T01:58:32.157299+0000","last_became_peered":"2024-03-26T01:58:32.157299+0000","last_unstale":"2024-03-26T01:58:59.182336+0000","last_undegraded":"2024-03-26T01:58:59.182336+0000","last_fullsized":"2024-03-26T01:58:59.182336+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:43:42.453770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183883+0000","last_change":"2024-03-26T01:58:36.183012+0000","last_active":"2024-03-26T01:58:59.183883+0000","last_peered":"2024-03-26T01:58:59.183883+0000","last_clean":"2024-03-26T01:58:59.183883+0000","last_became_active":"2024-03-26T01:58:36.182657+0000","last_became_peered":"2024-03-26T01:58:36.182657+0000","last_unstale":"2024-03-26T01:58:59.183883+0000","last_undegraded":"2024-03-26T01:58:59.183883+0000","last_fullsized":"2024-03-26T01:58:59.183883+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:05:25.331266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180748+0000","last_change":"2024-03-26T01:58:34.165372+0000","last_active":"2024-03-26T01:58:59.180748+0000","last_peered":"2024-03-26T01:58:59.180748+0000","last_clean":"2024-03-26T01:58:59.180748+0000","last_became_active":"2024-03-26T01:58:34.164197+0000","last_became_peered":"2024-03-26T01:58:34.164197+0000","last_unstale":"2024-03-26T01:58:59.180748+0000","last_undegraded":"2024-03-26T01:58:59.180748+0000","last_fullsized":"2024-03-26T01:58:59.180748+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:59:03.909926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.186999+0000","last_change":"2024-03-26T01:58:30.139473+0000","last_active":"2024-03-26T01:58:59.186999+0000","last_peered":"2024-03-26T01:58:59.186999+0000","last_clean":"2024-03-26T01:58:59.186999+0000","last_became_active":"2024-03-26T01:58:30.138715+0000","last_became_peered":"2024-03-26T01:58:30.138715+0000","last_unstale":"2024-03-26T01:58:59.186999+0000","last_undegraded":"2024-03-26T01:58:59.186999+0000","last_fullsized":"2024-03-26T01:58:59.186999+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:09:19.811105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181860+0000","last_change":"2024-03-26T01:58:32.159428+0000","last_active":"2024-03-26T01:58:59.181860+0000","last_peered":"2024-03-26T01:58:59.181860+0000","last_clean":"2024-03-26T01:58:59.181860+0000","last_became_active":"2024-03-26T01:58:32.159223+0000","last_became_peered":"2024-03-26T01:58:32.159223+0000","last_unstale":"2024-03-26T01:58:59.181860+0000","last_undegraded":"2024-03-26T01:58:59.181860+0000","last_fullsized":"2024-03-26T01:58:59.181860+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:28:08.287176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180635+0000","last_change":"2024-03-26T01:58:36.188560+0000","last_active":"2024-03-26T01:58:59.180635+0000","last_peered":"2024-03-26T01:58:59.180635+0000","last_clean":"2024-03-26T01:58:59.180635+0000","last_became_active":"2024-03-26T01:58:36.188193+0000","last_became_peered":"2024-03-26T01:58:36.188193+0000","last_unstale":"2024-03-26T01:58:59.180635+0000","last_undegraded":"2024-03-26T01:58:59.180635+0000","last_fullsized":"2024-03-26T01:58:59.180635+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:04:58.798057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180849+0000","last_change":"2024-03-26T01:58:34.167642+0000","last_active":"2024-03-26T01:58:59.180849+0000","last_peered":"2024-03-26T01:58:59.180849+0000","last_clean":"2024-03-26T01:58:59.180849+0000","last_became_active":"2024-03-26T01:58:34.167342+0000","last_became_peered":"2024-03-26T01:58:34.167342+0000","last_unstale":"2024-03-26T01:58:59.180849+0000","last_undegraded":"2024-03-26T01:58:59.180849+0000","last_fullsized":"2024-03-26T01:58:59.180849+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:07:09.662945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185142+0000","last_change":"2024-03-26T01:58:30.137539+0000","last_active":"2024-03-26T01:58:59.185142+0000","last_peered":"2024-03-26T01:58:59.185142+0000","last_clean":"2024-03-26T01:58:59.185142+0000","last_became_active":"2024-03-26T01:58:30.136866+0000","last_became_peered":"2024-03-26T01:58:30.136866+0000","last_unstale":"2024-03-26T01:58:59.185142+0000","last_undegraded":"2024-03-26T01:58:59.185142+0000","last_fullsized":"2024-03-26T01:58:59.185142+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:07:22.543441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"31'18","reported_seq":58,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.186441+0000","last_change":"2024-03-26T01:58:32.156713+0000","last_active":"2024-03-26T01:58:59.186441+0000","last_peered":"2024-03-26T01:58:59.186441+0000","last_clean":"2024-03-26T01:58:59.186441+0000","last_became_active":"2024-03-26T01:58:32.156303+0000","last_became_peered":"2024-03-26T01:58:32.156303+0000","last_unstale":"2024-03-26T01:58:59.186441+0000","last_undegraded":"2024-03-26T01:58:59.186441+0000","last_fullsized":"2024-03-26T01:58:59.186441+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:55:39.189779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181640+0000","last_change":"2024-03-26T01:58:36.186762+0000","last_active":"2024-03-26T01:58:59.181640+0000","last_peered":"2024-03-26T01:58:59.181640+0000","last_clean":"2024-03-26T01:58:59.181640+0000","last_became_active":"2024-03-26T01:58:36.185062+0000","last_became_peered":"2024-03-26T01:58:36.185062+0000","last_unstale":"2024-03-26T01:58:59.181640+0000","last_undegraded":"2024-03-26T01:58:59.181640+0000","last_fullsized":"2024-03-26T01:58:59.181640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:48:06.899232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185938+0000","last_change":"2024-03-26T01:58:34.165186+0000","last_active":"2024-03-26T01:58:59.185938+0000","last_peered":"2024-03-26T01:58:59.185938+0000","last_clean":"2024-03-26T01:58:59.185938+0000","last_became_active":"2024-03-26T01:58:34.164958+0000","last_became_peered":"2024-03-26T01:58:34.164958+0000","last_unstale":"2024-03-26T01:58:59.185938+0000","last_undegraded":"2024-03-26T01:58:59.185938+0000","last_fullsized":"2024-03-26T01:58:59.185938+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:28:54.247036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185766+0000","last_change":"2024-03-26T01:58:30.136301+0000","last_active":"2024-03-26T01:58:59.185766+0000","last_peered":"2024-03-26T01:58:59.185766+0000","last_clean":"2024-03-26T01:58:59.185766+0000","last_became_active":"2024-03-26T01:58:30.134461+0000","last_became_peered":"2024-03-26T01:58:30.134461+0000","last_unstale":"2024-03-26T01:58:59.185766+0000","last_undegraded":"2024-03-26T01:58:59.185766+0000","last_fullsized":"2024-03-26T01:58:59.185766+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:20:47.835309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"31'21","reported_seq":63,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184841+0000","last_change":"2024-03-26T01:58:32.158431+0000","last_active":"2024-03-26T01:58:59.184841+0000","last_peered":"2024-03-26T01:58:59.184841+0000","last_clean":"2024-03-26T01:58:59.184841+0000","last_became_active":"2024-03-26T01:58:32.158133+0000","last_became_peered":"2024-03-26T01:58:32.158133+0000","last_unstale":"2024-03-26T01:58:59.184841+0000","last_undegraded":"2024-03-26T01:58:59.184841+0000","last_fullsized":"2024-03-26T01:58:59.184841+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:28:27.108716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182341+0000","last_change":"2024-03-26T01:58:36.187900+0000","last_active":"2024-03-26T01:58:59.182341+0000","last_peered":"2024-03-26T01:58:59.182341+0000","last_clean":"2024-03-26T01:58:59.182341+0000","last_became_active":"2024-03-26T01:58:36.187576+0000","last_became_peered":"2024-03-26T01:58:36.187576+0000","last_unstale":"2024-03-26T01:58:59.182341+0000","last_undegraded":"2024-03-26T01:58:59.182341+0000","last_fullsized":"2024-03-26T01:58:59.182341+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:50:30.587849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185452+0000","last_change":"2024-03-26T01:58:34.171690+0000","last_active":"2024-03-26T01:58:59.185452+0000","last_peered":"2024-03-26T01:58:59.185452+0000","last_clean":"2024-03-26T01:58:59.185452+0000","last_became_active":"2024-03-26T01:58:34.171272+0000","last_became_peered":"2024-03-26T01:58:34.171272+0000","last_unstale":"2024-03-26T01:58:59.185452+0000","last_undegraded":"2024-03-26T01:58:59.185452+0000","last_fullsized":"2024-03-26T01:58:59.185452+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:26:12.716480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181154+0000","last_change":"2024-03-26T01:58:30.135343+0000","last_active":"2024-03-26T01:58:59.181154+0000","last_peered":"2024-03-26T01:58:59.181154+0000","last_clean":"2024-03-26T01:58:59.181154+0000","last_became_active":"2024-03-26T01:58:30.134604+0000","last_became_peered":"2024-03-26T01:58:30.134604+0000","last_unstale":"2024-03-26T01:58:59.181154+0000","last_undegraded":"2024-03-26T01:58:59.181154+0000","last_fullsized":"2024-03-26T01:58:59.181154+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:07:12.495786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'11","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185585+0000","last_change":"2024-03-26T01:58:32.157084+0000","last_active":"2024-03-26T01:58:59.185585+0000","last_peered":"2024-03-26T01:58:59.185585+0000","last_clean":"2024-03-26T01:58:59.185585+0000","last_became_active":"2024-03-26T01:58:32.156556+0000","last_became_peered":"2024-03-26T01:58:32.156556+0000","last_unstale":"2024-03-26T01:58:59.185585+0000","last_undegraded":"2024-03-26T01:58:59.185585+0000","last_fullsized":"2024-03-26T01:58:59.185585+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:58:48.935597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181749+0000","last_change":"2024-03-26T01:58:36.187742+0000","last_active":"2024-03-26T01:58:59.181749+0000","last_peered":"2024-03-26T01:58:59.181749+0000","last_clean":"2024-03-26T01:58:59.181749+0000","last_became_active":"2024-03-26T01:58:36.187374+0000","last_became_peered":"2024-03-26T01:58:36.187374+0000","last_unstale":"2024-03-26T01:58:59.181749+0000","last_undegraded":"2024-03-26T01:58:59.181749+0000","last_fullsized":"2024-03-26T01:58:59.181749+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:11:14.940926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"31'8","reported_seq":40,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:10.187213+0000","last_change":"2024-03-26T01:58:34.171888+0000","last_active":"2024-03-26T01:59:10.187213+0000","last_peered":"2024-03-26T01:59:10.187213+0000","last_clean":"2024-03-26T01:59:10.187213+0000","last_became_active":"2024-03-26T01:58:34.171283+0000","last_became_peered":"2024-03-26T01:58:34.171283+0000","last_unstale":"2024-03-26T01:59:10.187213+0000","last_undegraded":"2024-03-26T01:59:10.187213+0000","last_fullsized":"2024-03-26T01:59:10.187213+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:44:21.909014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179715+0000","last_change":"2024-03-26T01:58:30.136341+0000","last_active":"2024-03-26T01:58:59.179715+0000","last_peered":"2024-03-26T01:58:59.179715+0000","last_clean":"2024-03-26T01:58:59.179715+0000","last_became_active":"2024-03-26T01:58:30.136052+0000","last_became_peered":"2024-03-26T01:58:30.136052+0000","last_unstale":"2024-03-26T01:58:59.179715+0000","last_undegraded":"2024-03-26T01:58:59.179715+0000","last_fullsized":"2024-03-26T01:58:59.179715+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:06:48.642790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'15","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179138+0000","last_change":"2024-03-26T01:58:32.161289+0000","last_active":"2024-03-26T01:58:59.179138+0000","last_peered":"2024-03-26T01:58:59.179138+0000","last_clean":"2024-03-26T01:58:59.179138+0000","last_became_active":"2024-03-26T01:58:32.160902+0000","last_became_peered":"2024-03-26T01:58:32.160902+0000","last_unstale":"2024-03-26T01:58:59.179138+0000","last_undegraded":"2024-03-26T01:58:59.179138+0000","last_fullsized":"2024-03-26T01:58:59.179138+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:01:43.388216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181639+0000","last_change":"2024-03-26T01:58:36.188324+0000","last_active":"2024-03-26T01:58:59.181639+0000","last_peered":"2024-03-26T01:58:59.181639+0000","last_clean":"2024-03-26T01:58:59.181639+0000","last_became_active":"2024-03-26T01:58:36.187820+0000","last_became_peered":"2024-03-26T01:58:36.187820+0000","last_unstale":"2024-03-26T01:58:59.181639+0000","last_undegraded":"2024-03-26T01:58:59.181639+0000","last_fullsized":"2024-03-26T01:58:59.181639+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:40:49.955325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181910+0000","last_change":"2024-03-26T01:58:34.173274+0000","last_active":"2024-03-26T01:58:59.181910+0000","last_peered":"2024-03-26T01:58:59.181910+0000","last_clean":"2024-03-26T01:58:59.181910+0000","last_became_active":"2024-03-26T01:58:34.172532+0000","last_became_peered":"2024-03-26T01:58:34.172532+0000","last_unstale":"2024-03-26T01:58:59.181910+0000","last_undegraded":"2024-03-26T01:58:59.181910+0000","last_fullsized":"2024-03-26T01:58:59.181910+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:17:33.036512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183744+0000","last_change":"2024-03-26T01:58:30.139313+0000","last_active":"2024-03-26T01:58:59.183744+0000","last_peered":"2024-03-26T01:58:59.183744+0000","last_clean":"2024-03-26T01:58:59.183744+0000","last_became_active":"2024-03-26T01:58:30.138808+0000","last_became_peered":"2024-03-26T01:58:30.138808+0000","last_unstale":"2024-03-26T01:58:59.183744+0000","last_undegraded":"2024-03-26T01:58:59.183744+0000","last_fullsized":"2024-03-26T01:58:59.183744+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:06:28.430981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"31'41","reported_seq":98,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179720+0000","last_change":"2024-03-26T01:58:32.155029+0000","last_active":"2024-03-26T01:58:59.179720+0000","last_peered":"2024-03-26T01:58:59.179720+0000","last_clean":"2024-03-26T01:58:59.179720+0000","last_became_active":"2024-03-26T01:58:32.154326+0000","last_became_peered":"2024-03-26T01:58:32.154326+0000","last_unstale":"2024-03-26T01:58:59.179720+0000","last_undegraded":"2024-03-26T01:58:59.179720+0000","last_fullsized":"2024-03-26T01:58:59.179720+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:02:12.005363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180695+0000","last_change":"2024-03-26T01:58:36.185062+0000","last_active":"2024-03-26T01:58:59.180695+0000","last_peered":"2024-03-26T01:58:59.180695+0000","last_clean":"2024-03-26T01:58:59.180695+0000","last_became_active":"2024-03-26T01:58:36.184474+0000","last_became_peered":"2024-03-26T01:58:36.184474+0000","last_unstale":"2024-03-26T01:58:59.180695+0000","last_undegraded":"2024-03-26T01:58:59.180695+0000","last_fullsized":"2024-03-26T01:58:59.180695+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:13:33.577968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179208+0000","last_change":"2024-03-26T01:58:34.171576+0000","last_active":"2024-03-26T01:58:59.179208+0000","last_peered":"2024-03-26T01:58:59.179208+0000","last_clean":"2024-03-26T01:58:59.179208+0000","last_became_active":"2024-03-26T01:58:34.171107+0000","last_became_peered":"2024-03-26T01:58:34.171107+0000","last_unstale":"2024-03-26T01:58:59.179208+0000","last_undegraded":"2024-03-26T01:58:59.179208+0000","last_fullsized":"2024-03-26T01:58:59.179208+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:53:57.800083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183266+0000","last_change":"2024-03-26T01:58:30.140709+0000","last_active":"2024-03-26T01:58:59.183266+0000","last_peered":"2024-03-26T01:58:59.183266+0000","last_clean":"2024-03-26T01:58:59.183266+0000","last_became_active":"2024-03-26T01:58:30.140150+0000","last_became_peered":"2024-03-26T01:58:30.140150+0000","last_unstale":"2024-03-26T01:58:59.183266+0000","last_undegraded":"2024-03-26T01:58:59.183266+0000","last_fullsized":"2024-03-26T01:58:59.183266+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:45:05.542683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"31'21","reported_seq":63,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181998+0000","last_change":"2024-03-26T01:58:32.160893+0000","last_active":"2024-03-26T01:58:59.181998+0000","last_peered":"2024-03-26T01:58:59.181998+0000","last_clean":"2024-03-26T01:58:59.181998+0000","last_became_active":"2024-03-26T01:58:32.160597+0000","last_became_peered":"2024-03-26T01:58:32.160597+0000","last_unstale":"2024-03-26T01:58:59.181998+0000","last_undegraded":"2024-03-26T01:58:59.181998+0000","last_fullsized":"2024-03-26T01:58:59.181998+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:53:14.326937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180279+0000","last_change":"2024-03-26T01:58:36.188464+0000","last_active":"2024-03-26T01:58:59.180279+0000","last_peered":"2024-03-26T01:58:59.180279+0000","last_clean":"2024-03-26T01:58:59.180279+0000","last_became_active":"2024-03-26T01:58:36.187867+0000","last_became_peered":"2024-03-26T01:58:36.187867+0000","last_unstale":"2024-03-26T01:58:59.180279+0000","last_undegraded":"2024-03-26T01:58:59.180279+0000","last_fullsized":"2024-03-26T01:58:59.180279+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:10:34.963271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179427+0000","last_change":"2024-03-26T01:58:34.174164+0000","last_active":"2024-03-26T01:58:59.179427+0000","last_peered":"2024-03-26T01:58:59.179427+0000","last_clean":"2024-03-26T01:58:59.179427+0000","last_became_active":"2024-03-26T01:58:34.172669+0000","last_became_peered":"2024-03-26T01:58:34.172669+0000","last_unstale":"2024-03-26T01:58:59.179427+0000","last_undegraded":"2024-03-26T01:58:59.179427+0000","last_fullsized":"2024-03-26T01:58:59.179427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:30:04.294454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182145+0000","last_change":"2024-03-26T01:58:30.139862+0000","last_active":"2024-03-26T01:58:59.182145+0000","last_peered":"2024-03-26T01:58:59.182145+0000","last_clean":"2024-03-26T01:58:59.182145+0000","last_became_active":"2024-03-26T01:58:30.139467+0000","last_became_peered":"2024-03-26T01:58:30.139467+0000","last_unstale":"2024-03-26T01:58:59.182145+0000","last_undegraded":"2024-03-26T01:58:59.182145+0000","last_fullsized":"2024-03-26T01:58:59.182145+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:31:28.069553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"31'12","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182810+0000","last_change":"2024-03-26T01:58:32.157228+0000","last_active":"2024-03-26T01:58:59.182810+0000","last_peered":"2024-03-26T01:58:59.182810+0000","last_clean":"2024-03-26T01:58:59.182810+0000","last_became_active":"2024-03-26T01:58:32.155976+0000","last_became_peered":"2024-03-26T01:58:32.155976+0000","last_unstale":"2024-03-26T01:58:59.182810+0000","last_undegraded":"2024-03-26T01:58:59.182810+0000","last_fullsized":"2024-03-26T01:58:59.182810+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:04:28.437573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184111+0000","last_change":"2024-03-26T01:58:36.182835+0000","last_active":"2024-03-26T01:58:59.184111+0000","last_peered":"2024-03-26T01:58:59.184111+0000","last_clean":"2024-03-26T01:58:59.184111+0000","last_became_active":"2024-03-26T01:58:36.182517+0000","last_became_peered":"2024-03-26T01:58:36.182517+0000","last_unstale":"2024-03-26T01:58:59.184111+0000","last_undegraded":"2024-03-26T01:58:59.184111+0000","last_fullsized":"2024-03-26T01:58:59.184111+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:27:10.816704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180928+0000","last_change":"2024-03-26T01:58:34.169080+0000","last_active":"2024-03-26T01:58:59.180928+0000","last_peered":"2024-03-26T01:58:59.180928+0000","last_clean":"2024-03-26T01:58:59.180928+0000","last_became_active":"2024-03-26T01:58:34.167922+0000","last_became_peered":"2024-03-26T01:58:34.167922+0000","last_unstale":"2024-03-26T01:58:59.180928+0000","last_undegraded":"2024-03-26T01:58:59.180928+0000","last_fullsized":"2024-03-26T01:58:59.180928+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:47:35.604280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.186414+0000","last_change":"2024-03-26T01:58:30.139157+0000","last_active":"2024-03-26T01:58:59.186414+0000","last_peered":"2024-03-26T01:58:59.186414+0000","last_clean":"2024-03-26T01:58:59.186414+0000","last_became_active":"2024-03-26T01:58:30.137565+0000","last_became_peered":"2024-03-26T01:58:30.137565+0000","last_unstale":"2024-03-26T01:58:59.186414+0000","last_undegraded":"2024-03-26T01:58:59.186414+0000","last_fullsized":"2024-03-26T01:58:59.186414+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:25:29.921875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'24","reported_seq":64,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182832+0000","last_change":"2024-03-26T01:58:32.157693+0000","last_active":"2024-03-26T01:58:59.182832+0000","last_peered":"2024-03-26T01:58:59.182832+0000","last_clean":"2024-03-26T01:58:59.182832+0000","last_became_active":"2024-03-26T01:58:32.157400+0000","last_became_peered":"2024-03-26T01:58:32.157400+0000","last_unstale":"2024-03-26T01:58:59.182832+0000","last_undegraded":"2024-03-26T01:58:59.182832+0000","last_fullsized":"2024-03-26T01:58:59.182832+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:29:19.300415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185010+0000","last_change":"2024-03-26T01:58:36.183640+0000","last_active":"2024-03-26T01:58:59.185010+0000","last_peered":"2024-03-26T01:58:59.185010+0000","last_clean":"2024-03-26T01:58:59.185010+0000","last_became_active":"2024-03-26T01:58:36.182837+0000","last_became_peered":"2024-03-26T01:58:36.182837+0000","last_unstale":"2024-03-26T01:58:59.185010+0000","last_undegraded":"2024-03-26T01:58:59.185010+0000","last_fullsized":"2024-03-26T01:58:59.185010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:53:32.261663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179971+0000","last_change":"2024-03-26T01:58:34.171639+0000","last_active":"2024-03-26T01:58:59.179971+0000","last_peered":"2024-03-26T01:58:59.179971+0000","last_clean":"2024-03-26T01:58:59.179971+0000","last_became_active":"2024-03-26T01:58:34.171196+0000","last_became_peered":"2024-03-26T01:58:34.171196+0000","last_unstale":"2024-03-26T01:58:59.179971+0000","last_undegraded":"2024-03-26T01:58:59.179971+0000","last_fullsized":"2024-03-26T01:58:59.179971+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:59:38.446244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182248+0000","last_change":"2024-03-26T01:58:30.140212+0000","last_active":"2024-03-26T01:58:59.182248+0000","last_peered":"2024-03-26T01:58:59.182248+0000","last_clean":"2024-03-26T01:58:59.182248+0000","last_became_active":"2024-03-26T01:58:30.139663+0000","last_became_peered":"2024-03-26T01:58:30.139663+0000","last_unstale":"2024-03-26T01:58:59.182248+0000","last_undegraded":"2024-03-26T01:58:59.182248+0000","last_fullsized":"2024-03-26T01:58:59.182248+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:09:08.640306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"31'15","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.186169+0000","last_change":"2024-03-26T01:58:32.156307+0000","last_active":"2024-03-26T01:58:59.186169+0000","last_peered":"2024-03-26T01:58:59.186169+0000","last_clean":"2024-03-26T01:58:59.186169+0000","last_became_active":"2024-03-26T01:58:32.155809+0000","last_became_peered":"2024-03-26T01:58:32.155809+0000","last_unstale":"2024-03-26T01:58:59.186169+0000","last_undegraded":"2024-03-26T01:58:59.186169+0000","last_fullsized":"2024-03-26T01:58:59.186169+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:27:47.247203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180971+0000","last_change":"2024-03-26T01:58:36.183063+0000","last_active":"2024-03-26T01:58:59.180971+0000","last_peered":"2024-03-26T01:58:59.180971+0000","last_clean":"2024-03-26T01:58:59.180971+0000","last_became_active":"2024-03-26T01:58:36.182678+0000","last_became_peered":"2024-03-26T01:58:36.182678+0000","last_unstale":"2024-03-26T01:58:59.180971+0000","last_undegraded":"2024-03-26T01:58:59.180971+0000","last_fullsized":"2024-03-26T01:58:59.180971+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:44:45.389150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"31'8","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:10.187193+0000","last_change":"2024-03-26T01:58:34.168622+0000","last_active":"2024-03-26T01:59:10.187193+0000","last_peered":"2024-03-26T01:59:10.187193+0000","last_clean":"2024-03-26T01:59:10.187193+0000","last_became_active":"2024-03-26T01:58:34.168268+0000","last_became_peered":"2024-03-26T01:58:34.168268+0000","last_unstale":"2024-03-26T01:59:10.187193+0000","last_undegraded":"2024-03-26T01:59:10.187193+0000","last_fullsized":"2024-03-26T01:59:10.187193+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:50:32.236929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185492+0000","last_change":"2024-03-26T01:58:30.134776+0000","last_active":"2024-03-26T01:58:59.185492+0000","last_peered":"2024-03-26T01:58:59.185492+0000","last_clean":"2024-03-26T01:58:59.185492+0000","last_became_active":"2024-03-26T01:58:30.134358+0000","last_became_peered":"2024-03-26T01:58:30.134358+0000","last_unstale":"2024-03-26T01:58:59.185492+0000","last_undegraded":"2024-03-26T01:58:59.185492+0000","last_fullsized":"2024-03-26T01:58:59.185492+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:51:21.045204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180437+0000","last_change":"2024-03-26T01:58:32.157227+0000","last_active":"2024-03-26T01:58:59.180437+0000","last_peered":"2024-03-26T01:58:59.180437+0000","last_clean":"2024-03-26T01:58:59.180437+0000","last_became_active":"2024-03-26T01:58:32.155693+0000","last_became_peered":"2024-03-26T01:58:32.155693+0000","last_unstale":"2024-03-26T01:58:59.180437+0000","last_undegraded":"2024-03-26T01:58:59.180437+0000","last_fullsized":"2024-03-26T01:58:59.180437+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:10:38.354365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181521+0000","last_change":"2024-03-26T01:58:36.188246+0000","last_active":"2024-03-26T01:58:59.181521+0000","last_peered":"2024-03-26T01:58:59.181521+0000","last_clean":"2024-03-26T01:58:59.181521+0000","last_became_active":"2024-03-26T01:58:36.186544+0000","last_became_peered":"2024-03-26T01:58:36.186544+0000","last_unstale":"2024-03-26T01:58:59.181521+0000","last_undegraded":"2024-03-26T01:58:59.181521+0000","last_fullsized":"2024-03-26T01:58:59.181521+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:16:56.738069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'8","reported_seq":40,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:10.187239+0000","last_change":"2024-03-26T01:58:34.170342+0000","last_active":"2024-03-26T01:59:10.187239+0000","last_peered":"2024-03-26T01:59:10.187239+0000","last_clean":"2024-03-26T01:59:10.187239+0000","last_became_active":"2024-03-26T01:58:34.170021+0000","last_became_peered":"2024-03-26T01:58:34.170021+0000","last_unstale":"2024-03-26T01:59:10.187239+0000","last_undegraded":"2024-03-26T01:59:10.187239+0000","last_fullsized":"2024-03-26T01:59:10.187239+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:19:54.773039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184365+0000","last_change":"2024-03-26T01:58:30.134886+0000","last_active":"2024-03-26T01:58:59.184365+0000","last_peered":"2024-03-26T01:58:59.184365+0000","last_clean":"2024-03-26T01:58:59.184365+0000","last_became_active":"2024-03-26T01:58:30.134468+0000","last_became_peered":"2024-03-26T01:58:30.134468+0000","last_unstale":"2024-03-26T01:58:59.184365+0000","last_undegraded":"2024-03-26T01:58:59.184365+0000","last_fullsized":"2024-03-26T01:58:59.184365+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:25:41.189124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181146+0000","last_change":"2024-03-26T01:58:32.159576+0000","last_active":"2024-03-26T01:58:59.181146+0000","last_peered":"2024-03-26T01:58:59.181146+0000","last_clean":"2024-03-26T01:58:59.181146+0000","last_became_active":"2024-03-26T01:58:32.159279+0000","last_became_peered":"2024-03-26T01:58:32.159279+0000","last_unstale":"2024-03-26T01:58:59.181146+0000","last_undegraded":"2024-03-26T01:58:59.181146+0000","last_fullsized":"2024-03-26T01:58:59.181146+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:11:55.406190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181629+0000","last_change":"2024-03-26T01:58:36.186745+0000","last_active":"2024-03-26T01:58:59.181629+0000","last_peered":"2024-03-26T01:58:59.181629+0000","last_clean":"2024-03-26T01:58:59.181629+0000","last_became_active":"2024-03-26T01:58:36.186386+0000","last_became_peered":"2024-03-26T01:58:36.186386+0000","last_unstale":"2024-03-26T01:58:59.181629+0000","last_undegraded":"2024-03-26T01:58:59.181629+0000","last_fullsized":"2024-03-26T01:58:59.181629+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:37:35.511792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182144+0000","last_change":"2024-03-26T01:58:34.175304+0000","last_active":"2024-03-26T01:58:59.182144+0000","last_peered":"2024-03-26T01:58:59.182144+0000","last_clean":"2024-03-26T01:58:59.182144+0000","last_became_active":"2024-03-26T01:58:34.174740+0000","last_became_peered":"2024-03-26T01:58:34.174740+0000","last_unstale":"2024-03-26T01:58:59.182144+0000","last_undegraded":"2024-03-26T01:58:59.182144+0000","last_fullsized":"2024-03-26T01:58:59.182144+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:13:52.155008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180521+0000","last_change":"2024-03-26T01:58:30.139478+0000","last_active":"2024-03-26T01:58:59.180521+0000","last_peered":"2024-03-26T01:58:59.180521+0000","last_clean":"2024-03-26T01:58:59.180521+0000","last_became_active":"2024-03-26T01:58:30.139141+0000","last_became_peered":"2024-03-26T01:58:30.139141+0000","last_unstale":"2024-03-26T01:58:59.180521+0000","last_undegraded":"2024-03-26T01:58:59.180521+0000","last_fullsized":"2024-03-26T01:58:59.180521+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:15:09.336965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"31'4","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180228+0000","last_change":"2024-03-26T01:58:32.161117+0000","last_active":"2024-03-26T01:58:59.180228+0000","last_peered":"2024-03-26T01:58:59.180228+0000","last_clean":"2024-03-26T01:58:59.180228+0000","last_became_active":"2024-03-26T01:58:32.160778+0000","last_became_peered":"2024-03-26T01:58:32.160778+0000","last_unstale":"2024-03-26T01:58:59.180228+0000","last_undegraded":"2024-03-26T01:58:59.180228+0000","last_fullsized":"2024-03-26T01:58:59.180228+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:15:06.507400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181864+0000","last_change":"2024-03-26T01:58:36.190862+0000","last_active":"2024-03-26T01:58:59.181864+0000","last_peered":"2024-03-26T01:58:59.181864+0000","last_clean":"2024-03-26T01:58:59.181864+0000","last_became_active":"2024-03-26T01:58:36.190479+0000","last_became_peered":"2024-03-26T01:58:36.190479+0000","last_unstale":"2024-03-26T01:58:59.181864+0000","last_undegraded":"2024-03-26T01:58:59.181864+0000","last_fullsized":"2024-03-26T01:58:59.181864+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:06:41.524687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182006+0000","last_change":"2024-03-26T01:58:34.173699+0000","last_active":"2024-03-26T01:58:59.182006+0000","last_peered":"2024-03-26T01:58:59.182006+0000","last_clean":"2024-03-26T01:58:59.182006+0000","last_became_active":"2024-03-26T01:58:34.173342+0000","last_became_peered":"2024-03-26T01:58:34.173342+0000","last_unstale":"2024-03-26T01:58:59.182006+0000","last_undegraded":"2024-03-26T01:58:59.182006+0000","last_fullsized":"2024-03-26T01:58:59.182006+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:47:34.991903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183869+0000","last_change":"2024-03-26T01:58:30.138023+0000","last_active":"2024-03-26T01:58:59.183869+0000","last_peered":"2024-03-26T01:58:59.183869+0000","last_clean":"2024-03-26T01:58:59.183869+0000","last_became_active":"2024-03-26T01:58:30.137531+0000","last_became_peered":"2024-03-26T01:58:30.137531+0000","last_unstale":"2024-03-26T01:58:59.183869+0000","last_undegraded":"2024-03-26T01:58:59.183869+0000","last_fullsized":"2024-03-26T01:58:59.183869+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:08:47.173692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"31'17","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183870+0000","last_change":"2024-03-26T01:58:32.156330+0000","last_active":"2024-03-26T01:58:59.183870+0000","last_peered":"2024-03-26T01:58:59.183870+0000","last_clean":"2024-03-26T01:58:59.183870+0000","last_became_active":"2024-03-26T01:58:32.155812+0000","last_became_peered":"2024-03-26T01:58:32.155812+0000","last_unstale":"2024-03-26T01:58:59.183870+0000","last_undegraded":"2024-03-26T01:58:59.183870+0000","last_fullsized":"2024-03-26T01:58:59.183870+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:04:41.451735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184383+0000","last_change":"2024-03-26T01:58:36.190454+0000","last_active":"2024-03-26T01:58:59.184383+0000","last_peered":"2024-03-26T01:58:59.184383+0000","last_clean":"2024-03-26T01:58:59.184383+0000","last_became_active":"2024-03-26T01:58:36.190246+0000","last_became_peered":"2024-03-26T01:58:36.190246+0000","last_unstale":"2024-03-26T01:58:59.184383+0000","last_undegraded":"2024-03-26T01:58:59.184383+0000","last_fullsized":"2024-03-26T01:58:59.184383+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:45:37.113908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"24'2","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.186093+0000","last_change":"2024-03-26T01:58:30.134823+0000","last_active":"2024-03-26T01:58:59.186093+0000","last_peered":"2024-03-26T01:58:59.186093+0000","last_clean":"2024-03-26T01:58:59.186093+0000","last_became_active":"2024-03-26T01:58:30.134496+0000","last_became_peered":"2024-03-26T01:58:30.134496+0000","last_unstale":"2024-03-26T01:58:59.186093+0000","last_undegraded":"2024-03-26T01:58:59.186093+0000","last_fullsized":"2024-03-26T01:58:59.186093+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:58:42.368597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"31'17","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185367+0000","last_change":"2024-03-26T01:58:32.158350+0000","last_active":"2024-03-26T01:58:59.185367+0000","last_peered":"2024-03-26T01:58:59.185367+0000","last_clean":"2024-03-26T01:58:59.185367+0000","last_became_active":"2024-03-26T01:58:32.158043+0000","last_became_peered":"2024-03-26T01:58:32.158043+0000","last_unstale":"2024-03-26T01:58:59.185367+0000","last_undegraded":"2024-03-26T01:58:59.185367+0000","last_fullsized":"2024-03-26T01:58:59.185367+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:09:39.638621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'8","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:10.187229+0000","last_change":"2024-03-26T01:58:34.169259+0000","last_active":"2024-03-26T01:59:10.187229+0000","last_peered":"2024-03-26T01:59:10.187229+0000","last_clean":"2024-03-26T01:59:10.187229+0000","last_became_active":"2024-03-26T01:58:34.169027+0000","last_became_peered":"2024-03-26T01:58:34.169027+0000","last_unstale":"2024-03-26T01:59:10.187229+0000","last_undegraded":"2024-03-26T01:59:10.187229+0000","last_fullsized":"2024-03-26T01:59:10.187229+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:43:31.979307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180782+0000","last_change":"2024-03-26T01:58:36.182349+0000","last_active":"2024-03-26T01:58:59.180782+0000","last_peered":"2024-03-26T01:58:59.180782+0000","last_clean":"2024-03-26T01:58:59.180782+0000","last_became_active":"2024-03-26T01:58:36.180850+0000","last_became_peered":"2024-03-26T01:58:36.180850+0000","last_unstale":"2024-03-26T01:58:59.180782+0000","last_undegraded":"2024-03-26T01:58:59.180782+0000","last_fullsized":"2024-03-26T01:58:59.180782+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:18:30.428224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181471+0000","last_change":"2024-03-26T01:58:30.135532+0000","last_active":"2024-03-26T01:58:59.181471+0000","last_peered":"2024-03-26T01:58:59.181471+0000","last_clean":"2024-03-26T01:58:59.181471+0000","last_became_active":"2024-03-26T01:58:30.135209+0000","last_became_peered":"2024-03-26T01:58:30.135209+0000","last_unstale":"2024-03-26T01:58:59.181471+0000","last_undegraded":"2024-03-26T01:58:59.181471+0000","last_fullsized":"2024-03-26T01:58:59.181471+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:00:33.236148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"31'16","reported_seq":55,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:10.187166+0000","last_change":"2024-03-26T01:58:32.158315+0000","last_active":"2024-03-26T01:59:10.187166+0000","last_peered":"2024-03-26T01:59:10.187166+0000","last_clean":"2024-03-26T01:59:10.187166+0000","last_became_active":"2024-03-26T01:58:32.158073+0000","last_became_peered":"2024-03-26T01:58:32.158073+0000","last_unstale":"2024-03-26T01:59:10.187166+0000","last_undegraded":"2024-03-26T01:59:10.187166+0000","last_fullsized":"2024-03-26T01:59:10.187166+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:41:09.072781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184207+0000","last_change":"2024-03-26T01:58:34.168765+0000","last_active":"2024-03-26T01:58:59.184207+0000","last_peered":"2024-03-26T01:58:59.184207+0000","last_clean":"2024-03-26T01:58:59.184207+0000","last_became_active":"2024-03-26T01:58:34.168408+0000","last_became_peered":"2024-03-26T01:58:34.168408+0000","last_unstale":"2024-03-26T01:58:59.184207+0000","last_undegraded":"2024-03-26T01:58:59.184207+0000","last_fullsized":"2024-03-26T01:58:59.184207+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:45:39.608232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181542+0000","last_change":"2024-03-26T01:58:36.187579+0000","last_active":"2024-03-26T01:58:59.181542+0000","last_peered":"2024-03-26T01:58:59.181542+0000","last_clean":"2024-03-26T01:58:59.181542+0000","last_became_active":"2024-03-26T01:58:36.187213+0000","last_became_peered":"2024-03-26T01:58:36.187213+0000","last_unstale":"2024-03-26T01:58:59.181542+0000","last_undegraded":"2024-03-26T01:58:59.181542+0000","last_fullsized":"2024-03-26T01:58:59.181542+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:32:29.233621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181247+0000","last_change":"2024-03-26T01:58:30.140416+0000","last_active":"2024-03-26T01:58:59.181247+0000","last_peered":"2024-03-26T01:58:59.181247+0000","last_clean":"2024-03-26T01:58:59.181247+0000","last_became_active":"2024-03-26T01:58:30.140021+0000","last_became_peered":"2024-03-26T01:58:30.140021+0000","last_unstale":"2024-03-26T01:58:59.181247+0000","last_undegraded":"2024-03-26T01:58:59.181247+0000","last_fullsized":"2024-03-26T01:58:59.181247+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:27:40.235330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"31'25","reported_seq":71,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181174+0000","last_change":"2024-03-26T01:58:32.159326+0000","last_active":"2024-03-26T01:58:59.181174+0000","last_peered":"2024-03-26T01:58:59.181174+0000","last_clean":"2024-03-26T01:58:59.181174+0000","last_became_active":"2024-03-26T01:58:32.158986+0000","last_became_peered":"2024-03-26T01:58:32.158986+0000","last_unstale":"2024-03-26T01:58:59.181174+0000","last_undegraded":"2024-03-26T01:58:59.181174+0000","last_fullsized":"2024-03-26T01:58:59.181174+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:50:09.892516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181019+0000","last_change":"2024-03-26T01:58:34.172969+0000","last_active":"2024-03-26T01:58:59.181019+0000","last_peered":"2024-03-26T01:58:59.181019+0000","last_clean":"2024-03-26T01:58:59.181019+0000","last_became_active":"2024-03-26T01:58:34.172456+0000","last_became_peered":"2024-03-26T01:58:34.172456+0000","last_unstale":"2024-03-26T01:58:59.181019+0000","last_undegraded":"2024-03-26T01:58:59.181019+0000","last_fullsized":"2024-03-26T01:58:59.181019+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:55:25.288709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182673+0000","last_change":"2024-03-26T01:58:36.191043+0000","last_active":"2024-03-26T01:58:59.182673+0000","last_peered":"2024-03-26T01:58:59.182673+0000","last_clean":"2024-03-26T01:58:59.182673+0000","last_became_active":"2024-03-26T01:58:36.190609+0000","last_became_peered":"2024-03-26T01:58:36.190609+0000","last_unstale":"2024-03-26T01:58:59.182673+0000","last_undegraded":"2024-03-26T01:58:59.182673+0000","last_fullsized":"2024-03-26T01:58:59.182673+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:04:31.833880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"31'30","reported_seq":79,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183486+0000","last_change":"2024-03-26T01:58:32.158264+0000","last_active":"2024-03-26T01:58:59.183486+0000","last_peered":"2024-03-26T01:58:59.183486+0000","last_clean":"2024-03-26T01:58:59.183486+0000","last_became_active":"2024-03-26T01:58:32.157952+0000","last_became_peered":"2024-03-26T01:58:32.157952+0000","last_unstale":"2024-03-26T01:58:59.183486+0000","last_undegraded":"2024-03-26T01:58:59.183486+0000","last_fullsized":"2024-03-26T01:58:59.183486+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:19:27.115554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179608+0000","last_change":"2024-03-26T01:58:30.136558+0000","last_active":"2024-03-26T01:58:59.179608+0000","last_peered":"2024-03-26T01:58:59.179608+0000","last_clean":"2024-03-26T01:58:59.179608+0000","last_became_active":"2024-03-26T01:58:30.135134+0000","last_became_peered":"2024-03-26T01:58:30.135134+0000","last_unstale":"2024-03-26T01:58:59.179608+0000","last_undegraded":"2024-03-26T01:58:59.179608+0000","last_fullsized":"2024-03-26T01:58:59.179608+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:49:56.542251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182727+0000","last_change":"2024-03-26T01:58:34.174621+0000","last_active":"2024-03-26T01:58:59.182727+0000","last_peered":"2024-03-26T01:58:59.182727+0000","last_clean":"2024-03-26T01:58:59.182727+0000","last_became_active":"2024-03-26T01:58:34.173408+0000","last_became_peered":"2024-03-26T01:58:34.173408+0000","last_unstale":"2024-03-26T01:58:59.182727+0000","last_undegraded":"2024-03-26T01:58:59.182727+0000","last_fullsized":"2024-03-26T01:58:59.182727+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:27:47.938754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184612+0000","last_change":"2024-03-26T01:58:36.182933+0000","last_active":"2024-03-26T01:58:59.184612+0000","last_peered":"2024-03-26T01:58:59.184612+0000","last_clean":"2024-03-26T01:58:59.184612+0000","last_became_active":"2024-03-26T01:58:36.182576+0000","last_became_peered":"2024-03-26T01:58:36.182576+0000","last_unstale":"2024-03-26T01:58:59.184612+0000","last_undegraded":"2024-03-26T01:58:59.184612+0000","last_fullsized":"2024-03-26T01:58:59.184612+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:25:11.397494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"31'16","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180340+0000","last_change":"2024-03-26T01:58:32.156762+0000","last_active":"2024-03-26T01:58:59.180340+0000","last_peered":"2024-03-26T01:58:59.180340+0000","last_clean":"2024-03-26T01:58:59.180340+0000","last_became_active":"2024-03-26T01:58:32.155408+0000","last_became_peered":"2024-03-26T01:58:32.155408+0000","last_unstale":"2024-03-26T01:58:59.180340+0000","last_undegraded":"2024-03-26T01:58:59.180340+0000","last_fullsized":"2024-03-26T01:58:59.180340+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:47:27.517461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181334+0000","last_change":"2024-03-26T01:58:30.135224+0000","last_active":"2024-03-26T01:58:59.181334+0000","last_peered":"2024-03-26T01:58:59.181334+0000","last_clean":"2024-03-26T01:58:59.181334+0000","last_became_active":"2024-03-26T01:58:30.134183+0000","last_became_peered":"2024-03-26T01:58:30.134183+0000","last_unstale":"2024-03-26T01:58:59.181334+0000","last_undegraded":"2024-03-26T01:58:59.181334+0000","last_fullsized":"2024-03-26T01:58:59.181334+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:46:24.952993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"32'208","reported_seq":314,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:01.277658+0000","last_change":"2024-03-26T01:58:20.077262+0000","last_active":"2024-03-26T01:59:01.277658+0000","last_peered":"2024-03-26T01:59:01.277658+0000","last_clean":"2024-03-26T01:59:01.277658+0000","last_became_active":"2024-03-26T01:58:20.076923+0000","last_became_peered":"2024-03-26T01:58:20.076923+0000","last_unstale":"2024-03-26T01:59:01.277658+0000","last_undegraded":"2024-03-26T01:59:01.277658+0000","last_fullsized":"2024-03-26T01:59:01.277658+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:19.055798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:19.055798+0000","last_clean_scrub_stamp":"2024-03-26T01:58:19.055798+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:52:02.430554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181954+0000","last_change":"2024-03-26T01:58:34.167011+0000","last_active":"2024-03-26T01:58:59.181954+0000","last_peered":"2024-03-26T01:58:59.181954+0000","last_clean":"2024-03-26T01:58:59.181954+0000","last_became_active":"2024-03-26T01:58:34.166630+0000","last_became_peered":"2024-03-26T01:58:34.166630+0000","last_unstale":"2024-03-26T01:58:59.181954+0000","last_undegraded":"2024-03-26T01:58:59.181954+0000","last_fullsized":"2024-03-26T01:58:59.181954+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:40:17.454107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181747+0000","last_change":"2024-03-26T01:58:36.187993+0000","last_active":"2024-03-26T01:58:59.181747+0000","last_peered":"2024-03-26T01:58:59.181747+0000","last_clean":"2024-03-26T01:58:59.181747+0000","last_became_active":"2024-03-26T01:58:36.186411+0000","last_became_peered":"2024-03-26T01:58:36.186411+0000","last_unstale":"2024-03-26T01:58:59.181747+0000","last_undegraded":"2024-03-26T01:58:59.181747+0000","last_fullsized":"2024-03-26T01:58:59.181747+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:40:49.335258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"31'20","reported_seq":56,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179835+0000","last_change":"2024-03-26T01:58:32.156747+0000","last_active":"2024-03-26T01:58:59.179835+0000","last_peered":"2024-03-26T01:58:59.179835+0000","last_clean":"2024-03-26T01:58:59.179835+0000","last_became_active":"2024-03-26T01:58:32.155645+0000","last_became_peered":"2024-03-26T01:58:32.155645+0000","last_unstale":"2024-03-26T01:58:59.179835+0000","last_undegraded":"2024-03-26T01:58:59.179835+0000","last_fullsized":"2024-03-26T01:58:59.179835+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:07:21.232257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180114+0000","last_change":"2024-03-26T01:58:30.135446+0000","last_active":"2024-03-26T01:58:59.180114+0000","last_peered":"2024-03-26T01:58:59.180114+0000","last_clean":"2024-03-26T01:58:59.180114+0000","last_became_active":"2024-03-26T01:58:30.134026+0000","last_became_peered":"2024-03-26T01:58:30.134026+0000","last_unstale":"2024-03-26T01:58:59.180114+0000","last_undegraded":"2024-03-26T01:58:59.180114+0000","last_fullsized":"2024-03-26T01:58:59.180114+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:31:00.260417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184415+0000","last_change":"2024-03-26T01:58:34.171893+0000","last_active":"2024-03-26T01:58:59.184415+0000","last_peered":"2024-03-26T01:58:59.184415+0000","last_clean":"2024-03-26T01:58:59.184415+0000","last_became_active":"2024-03-26T01:58:34.171607+0000","last_became_peered":"2024-03-26T01:58:34.171607+0000","last_unstale":"2024-03-26T01:58:59.184415+0000","last_undegraded":"2024-03-26T01:58:59.184415+0000","last_fullsized":"2024-03-26T01:58:59.184415+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:36:47.600932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181055+0000","last_change":"2024-03-26T01:58:36.186337+0000","last_active":"2024-03-26T01:58:59.181055+0000","last_peered":"2024-03-26T01:58:59.181055+0000","last_clean":"2024-03-26T01:58:59.181055+0000","last_became_active":"2024-03-26T01:58:36.185156+0000","last_became_peered":"2024-03-26T01:58:36.185156+0000","last_unstale":"2024-03-26T01:58:59.181055+0000","last_undegraded":"2024-03-26T01:58:59.181055+0000","last_fullsized":"2024-03-26T01:58:59.181055+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:39:30.381455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.186270+0000","last_change":"2024-03-26T01:58:30.139116+0000","last_active":"2024-03-26T01:58:59.186270+0000","last_peered":"2024-03-26T01:58:59.186270+0000","last_clean":"2024-03-26T01:58:59.186270+0000","last_became_active":"2024-03-26T01:58:30.137541+0000","last_became_peered":"2024-03-26T01:58:30.137541+0000","last_unstale":"2024-03-26T01:58:59.186270+0000","last_undegraded":"2024-03-26T01:58:59.186270+0000","last_fullsized":"2024-03-26T01:58:59.186270+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:02:15.121649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'34","reported_seq":93,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:10.186844+0000","last_change":"2024-03-26T01:58:32.154799+0000","last_active":"2024-03-26T01:59:10.186844+0000","last_peered":"2024-03-26T01:59:10.186844+0000","last_clean":"2024-03-26T01:59:10.186844+0000","last_became_active":"2024-03-26T01:58:32.154338+0000","last_became_peered":"2024-03-26T01:58:32.154338+0000","last_unstale":"2024-03-26T01:59:10.186844+0000","last_undegraded":"2024-03-26T01:59:10.186844+0000","last_fullsized":"2024-03-26T01:59:10.186844+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:57:29.818605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"31'8","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:10.187165+0000","last_change":"2024-03-26T01:58:34.168792+0000","last_active":"2024-03-26T01:59:10.187165+0000","last_peered":"2024-03-26T01:59:10.187165+0000","last_clean":"2024-03-26T01:59:10.187165+0000","last_became_active":"2024-03-26T01:58:34.168445+0000","last_became_peered":"2024-03-26T01:58:34.168445+0000","last_unstale":"2024-03-26T01:59:10.187165+0000","last_undegraded":"2024-03-26T01:59:10.187165+0000","last_fullsized":"2024-03-26T01:59:10.187165+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:47:49.238420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179928+0000","last_change":"2024-03-26T01:58:36.188257+0000","last_active":"2024-03-26T01:58:59.179928+0000","last_peered":"2024-03-26T01:58:59.179928+0000","last_clean":"2024-03-26T01:58:59.179928+0000","last_became_active":"2024-03-26T01:58:36.186442+0000","last_became_peered":"2024-03-26T01:58:36.186442+0000","last_unstale":"2024-03-26T01:58:59.179928+0000","last_undegraded":"2024-03-26T01:58:59.179928+0000","last_fullsized":"2024-03-26T01:58:59.179928+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:17:03.649301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182228+0000","last_change":"2024-03-26T01:58:30.136657+0000","last_active":"2024-03-26T01:58:59.182228+0000","last_peered":"2024-03-26T01:58:59.182228+0000","last_clean":"2024-03-26T01:58:59.182228+0000","last_became_active":"2024-03-26T01:58:30.135563+0000","last_became_peered":"2024-03-26T01:58:30.135563+0000","last_unstale":"2024-03-26T01:58:59.182228+0000","last_undegraded":"2024-03-26T01:58:59.182228+0000","last_fullsized":"2024-03-26T01:58:59.182228+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:30:56.449784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"31'19","reported_seq":62,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183016+0000","last_change":"2024-03-26T01:58:32.157561+0000","last_active":"2024-03-26T01:58:59.183016+0000","last_peered":"2024-03-26T01:58:59.183016+0000","last_clean":"2024-03-26T01:58:59.183016+0000","last_became_active":"2024-03-26T01:58:32.156539+0000","last_became_peered":"2024-03-26T01:58:32.156539+0000","last_unstale":"2024-03-26T01:58:59.183016+0000","last_undegraded":"2024-03-26T01:58:59.183016+0000","last_fullsized":"2024-03-26T01:58:59.183016+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:52:01.988500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"31'8","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:10.186764+0000","last_change":"2024-03-26T01:58:34.175092+0000","last_active":"2024-03-26T01:59:10.186764+0000","last_peered":"2024-03-26T01:59:10.186764+0000","last_clean":"2024-03-26T01:59:10.186764+0000","last_became_active":"2024-03-26T01:58:34.174745+0000","last_became_peered":"2024-03-26T01:58:34.174745+0000","last_unstale":"2024-03-26T01:59:10.186764+0000","last_undegraded":"2024-03-26T01:59:10.186764+0000","last_fullsized":"2024-03-26T01:59:10.186764+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:39:28.143023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184283+0000","last_change":"2024-03-26T01:58:36.189161+0000","last_active":"2024-03-26T01:58:59.184283+0000","last_peered":"2024-03-26T01:58:59.184283+0000","last_clean":"2024-03-26T01:58:59.184283+0000","last_became_active":"2024-03-26T01:58:36.188783+0000","last_became_peered":"2024-03-26T01:58:36.188783+0000","last_unstale":"2024-03-26T01:58:59.184283+0000","last_undegraded":"2024-03-26T01:58:59.184283+0000","last_fullsized":"2024-03-26T01:58:59.184283+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:39:11.216002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183015+0000","last_change":"2024-03-26T01:58:30.140194+0000","last_active":"2024-03-26T01:58:59.183015+0000","last_peered":"2024-03-26T01:58:59.183015+0000","last_clean":"2024-03-26T01:58:59.183015+0000","last_became_active":"2024-03-26T01:58:30.139697+0000","last_became_peered":"2024-03-26T01:58:30.139697+0000","last_unstale":"2024-03-26T01:58:59.183015+0000","last_undegraded":"2024-03-26T01:58:59.183015+0000","last_fullsized":"2024-03-26T01:58:59.183015+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:41:19.704783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"31'12","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182348+0000","last_change":"2024-03-26T01:58:32.157225+0000","last_active":"2024-03-26T01:58:59.182348+0000","last_peered":"2024-03-26T01:58:59.182348+0000","last_clean":"2024-03-26T01:58:59.182348+0000","last_became_active":"2024-03-26T01:58:32.156161+0000","last_became_peered":"2024-03-26T01:58:32.156161+0000","last_unstale":"2024-03-26T01:58:59.182348+0000","last_undegraded":"2024-03-26T01:58:59.182348+0000","last_fullsized":"2024-03-26T01:58:59.182348+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:28:30.498738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183686+0000","last_change":"2024-03-26T01:58:34.168744+0000","last_active":"2024-03-26T01:58:59.183686+0000","last_peered":"2024-03-26T01:58:59.183686+0000","last_clean":"2024-03-26T01:58:59.183686+0000","last_became_active":"2024-03-26T01:58:34.168398+0000","last_became_peered":"2024-03-26T01:58:34.168398+0000","last_unstale":"2024-03-26T01:58:59.183686+0000","last_undegraded":"2024-03-26T01:58:59.183686+0000","last_fullsized":"2024-03-26T01:58:59.183686+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:49:37.806127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181911+0000","last_change":"2024-03-26T01:58:36.188194+0000","last_active":"2024-03-26T01:58:59.181911+0000","last_peered":"2024-03-26T01:58:59.181911+0000","last_clean":"2024-03-26T01:58:59.181911+0000","last_became_active":"2024-03-26T01:58:36.186464+0000","last_became_peered":"2024-03-26T01:58:36.186464+0000","last_unstale":"2024-03-26T01:58:59.181911+0000","last_undegraded":"2024-03-26T01:58:59.181911+0000","last_fullsized":"2024-03-26T01:58:59.181911+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:52:58.024240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"31'8","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:10.187025+0000","last_change":"2024-03-26T01:58:34.171383+0000","last_active":"2024-03-26T01:59:10.187025+0000","last_peered":"2024-03-26T01:59:10.187025+0000","last_clean":"2024-03-26T01:59:10.187025+0000","last_became_active":"2024-03-26T01:58:34.170560+0000","last_became_peered":"2024-03-26T01:58:34.170560+0000","last_unstale":"2024-03-26T01:59:10.187025+0000","last_undegraded":"2024-03-26T01:59:10.187025+0000","last_fullsized":"2024-03-26T01:59:10.187025+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:15:55.206007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179953+0000","last_change":"2024-03-26T01:58:30.135491+0000","last_active":"2024-03-26T01:58:59.179953+0000","last_peered":"2024-03-26T01:58:59.179953+0000","last_clean":"2024-03-26T01:58:59.179953+0000","last_became_active":"2024-03-26T01:58:30.134958+0000","last_became_peered":"2024-03-26T01:58:30.134958+0000","last_unstale":"2024-03-26T01:58:59.179953+0000","last_undegraded":"2024-03-26T01:58:59.179953+0000","last_fullsized":"2024-03-26T01:58:59.179953+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:03:32.726569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"31'24","reported_seq":70,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184139+0000","last_change":"2024-03-26T01:58:32.157117+0000","last_active":"2024-03-26T01:58:59.184139+0000","last_peered":"2024-03-26T01:58:59.184139+0000","last_clean":"2024-03-26T01:58:59.184139+0000","last_became_active":"2024-03-26T01:58:32.156788+0000","last_became_peered":"2024-03-26T01:58:32.156788+0000","last_unstale":"2024-03-26T01:58:59.184139+0000","last_undegraded":"2024-03-26T01:58:59.184139+0000","last_fullsized":"2024-03-26T01:58:59.184139+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:17:29.181591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184537+0000","last_change":"2024-03-26T01:58:36.183880+0000","last_active":"2024-03-26T01:58:59.184537+0000","last_peered":"2024-03-26T01:58:59.184537+0000","last_clean":"2024-03-26T01:58:59.184537+0000","last_became_active":"2024-03-26T01:58:36.183518+0000","last_became_peered":"2024-03-26T01:58:36.183518+0000","last_unstale":"2024-03-26T01:58:59.184537+0000","last_undegraded":"2024-03-26T01:58:59.184537+0000","last_fullsized":"2024-03-26T01:58:59.184537+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:43:45.259076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182232+0000","last_change":"2024-03-26T01:58:34.174097+0000","last_active":"2024-03-26T01:58:59.182232+0000","last_peered":"2024-03-26T01:58:59.182232+0000","last_clean":"2024-03-26T01:58:59.182232+0000","last_became_active":"2024-03-26T01:58:34.173258+0000","last_became_peered":"2024-03-26T01:58:34.173258+0000","last_unstale":"2024-03-26T01:58:59.182232+0000","last_undegraded":"2024-03-26T01:58:59.182232+0000","last_fullsized":"2024-03-26T01:58:59.182232+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:24:35.674377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179664+0000","last_change":"2024-03-26T01:58:30.135467+0000","last_active":"2024-03-26T01:58:59.179664+0000","last_peered":"2024-03-26T01:58:59.179664+0000","last_clean":"2024-03-26T01:58:59.179664+0000","last_became_active":"2024-03-26T01:58:30.135083+0000","last_became_peered":"2024-03-26T01:58:30.135083+0000","last_unstale":"2024-03-26T01:58:59.179664+0000","last_undegraded":"2024-03-26T01:58:59.179664+0000","last_fullsized":"2024-03-26T01:58:59.179664+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:45:14.791345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'28","reported_seq":73,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183860+0000","last_change":"2024-03-26T01:58:32.157107+0000","last_active":"2024-03-26T01:58:59.183860+0000","last_peered":"2024-03-26T01:58:59.183860+0000","last_clean":"2024-03-26T01:58:59.183860+0000","last_became_active":"2024-03-26T01:58:32.156812+0000","last_became_peered":"2024-03-26T01:58:32.156812+0000","last_unstale":"2024-03-26T01:58:59.183860+0000","last_undegraded":"2024-03-26T01:58:59.183860+0000","last_fullsized":"2024-03-26T01:58:59.183860+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:01:12.654416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185909+0000","last_change":"2024-03-26T01:58:36.183732+0000","last_active":"2024-03-26T01:58:59.185909+0000","last_peered":"2024-03-26T01:58:59.185909+0000","last_clean":"2024-03-26T01:58:59.185909+0000","last_became_active":"2024-03-26T01:58:36.183404+0000","last_became_peered":"2024-03-26T01:58:36.183404+0000","last_unstale":"2024-03-26T01:58:59.185909+0000","last_undegraded":"2024-03-26T01:58:59.185909+0000","last_fullsized":"2024-03-26T01:58:59.185909+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:54:49.610997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184825+0000","last_change":"2024-03-26T01:58:30.134700+0000","last_active":"2024-03-26T01:58:59.184825+0000","last_peered":"2024-03-26T01:58:59.184825+0000","last_clean":"2024-03-26T01:58:59.184825+0000","last_became_active":"2024-03-26T01:58:30.134336+0000","last_became_peered":"2024-03-26T01:58:30.134336+0000","last_unstale":"2024-03-26T01:58:59.184825+0000","last_undegraded":"2024-03-26T01:58:59.184825+0000","last_fullsized":"2024-03-26T01:58:59.184825+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:49:13.401189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"31'15","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181251+0000","last_change":"2024-03-26T01:58:32.156397+0000","last_active":"2024-03-26T01:58:59.181251+0000","last_peered":"2024-03-26T01:58:59.181251+0000","last_clean":"2024-03-26T01:58:59.181251+0000","last_became_active":"2024-03-26T01:58:32.155515+0000","last_became_peered":"2024-03-26T01:58:32.155515+0000","last_unstale":"2024-03-26T01:58:59.181251+0000","last_undegraded":"2024-03-26T01:58:59.181251+0000","last_fullsized":"2024-03-26T01:58:59.181251+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:30:51.951393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181761+0000","last_change":"2024-03-26T01:58:34.170142+0000","last_active":"2024-03-26T01:58:59.181761+0000","last_peered":"2024-03-26T01:58:59.181761+0000","last_clean":"2024-03-26T01:58:59.181761+0000","last_became_active":"2024-03-26T01:58:34.169668+0000","last_became_peered":"2024-03-26T01:58:34.169668+0000","last_unstale":"2024-03-26T01:58:59.181761+0000","last_undegraded":"2024-03-26T01:58:59.181761+0000","last_fullsized":"2024-03-26T01:58:59.181761+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:58:32.525102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180922+0000","last_change":"2024-03-26T01:58:36.183224+0000","last_active":"2024-03-26T01:58:59.180922+0000","last_peered":"2024-03-26T01:58:59.180922+0000","last_clean":"2024-03-26T01:58:59.180922+0000","last_became_active":"2024-03-26T01:58:36.182210+0000","last_became_peered":"2024-03-26T01:58:36.182210+0000","last_unstale":"2024-03-26T01:58:59.180922+0000","last_undegraded":"2024-03-26T01:58:59.180922+0000","last_fullsized":"2024-03-26T01:58:59.180922+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:40:42.882517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"31'8","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:59:10.187105+0000","last_change":"2024-03-26T01:58:34.167283+0000","last_active":"2024-03-26T01:59:10.187105+0000","last_peered":"2024-03-26T01:59:10.187105+0000","last_clean":"2024-03-26T01:59:10.187105+0000","last_became_active":"2024-03-26T01:58:34.166709+0000","last_became_peered":"2024-03-26T01:58:34.166709+0000","last_unstale":"2024-03-26T01:59:10.187105+0000","last_undegraded":"2024-03-26T01:59:10.187105+0000","last_fullsized":"2024-03-26T01:59:10.187105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:03:24.270177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185647+0000","last_change":"2024-03-26T01:58:30.136006+0000","last_active":"2024-03-26T01:58:59.185647+0000","last_peered":"2024-03-26T01:58:59.185647+0000","last_clean":"2024-03-26T01:58:59.185647+0000","last_became_active":"2024-03-26T01:58:30.135630+0000","last_became_peered":"2024-03-26T01:58:30.135630+0000","last_unstale":"2024-03-26T01:58:59.185647+0000","last_undegraded":"2024-03-26T01:58:59.185647+0000","last_fullsized":"2024-03-26T01:58:59.185647+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:28:02.175466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"31'25","reported_seq":66,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.181034+0000","last_change":"2024-03-26T01:58:32.160879+0000","last_active":"2024-03-26T01:58:59.181034+0000","last_peered":"2024-03-26T01:58:59.181034+0000","last_clean":"2024-03-26T01:58:59.181034+0000","last_became_active":"2024-03-26T01:58:32.160577+0000","last_became_peered":"2024-03-26T01:58:32.160577+0000","last_unstale":"2024-03-26T01:58:59.181034+0000","last_undegraded":"2024-03-26T01:58:59.181034+0000","last_fullsized":"2024-03-26T01:58:59.181034+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:46:59.282899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185138+0000","last_change":"2024-03-26T01:58:36.183872+0000","last_active":"2024-03-26T01:58:59.185138+0000","last_peered":"2024-03-26T01:58:59.185138+0000","last_clean":"2024-03-26T01:58:59.185138+0000","last_became_active":"2024-03-26T01:58:36.183540+0000","last_became_peered":"2024-03-26T01:58:36.183540+0000","last_unstale":"2024-03-26T01:58:59.185138+0000","last_undegraded":"2024-03-26T01:58:59.185138+0000","last_fullsized":"2024-03-26T01:58:59.185138+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:00:46.373628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.185409+0000","last_change":"2024-03-26T01:58:30.137193+0000","last_active":"2024-03-26T01:58:59.185409+0000","last_peered":"2024-03-26T01:58:59.185409+0000","last_clean":"2024-03-26T01:58:59.185409+0000","last_became_active":"2024-03-26T01:58:30.135632+0000","last_became_peered":"2024-03-26T01:58:30.135632+0000","last_unstale":"2024-03-26T01:58:59.185409+0000","last_undegraded":"2024-03-26T01:58:59.185409+0000","last_fullsized":"2024-03-26T01:58:59.185409+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:41:10.153458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'17","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182950+0000","last_change":"2024-03-26T01:58:32.157715+0000","last_active":"2024-03-26T01:58:59.182950+0000","last_peered":"2024-03-26T01:58:59.182950+0000","last_clean":"2024-03-26T01:58:59.182950+0000","last_became_active":"2024-03-26T01:58:32.157410+0000","last_became_peered":"2024-03-26T01:58:32.157410+0000","last_unstale":"2024-03-26T01:58:59.182950+0000","last_undegraded":"2024-03-26T01:58:59.182950+0000","last_fullsized":"2024-03-26T01:58:59.182950+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:07:07.497045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180964+0000","last_change":"2024-03-26T01:58:34.165811+0000","last_active":"2024-03-26T01:58:59.180964+0000","last_peered":"2024-03-26T01:58:59.180964+0000","last_clean":"2024-03-26T01:58:59.180964+0000","last_became_active":"2024-03-26T01:58:34.165511+0000","last_became_peered":"2024-03-26T01:58:34.165511+0000","last_unstale":"2024-03-26T01:58:59.180964+0000","last_undegraded":"2024-03-26T01:58:59.180964+0000","last_fullsized":"2024-03-26T01:58:59.180964+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:13:25.736549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.179952+0000","last_change":"2024-03-26T01:58:36.183039+0000","last_active":"2024-03-26T01:58:59.179952+0000","last_peered":"2024-03-26T01:58:59.179952+0000","last_clean":"2024-03-26T01:58:59.179952+0000","last_became_active":"2024-03-26T01:58:36.182662+0000","last_became_peered":"2024-03-26T01:58:36.182662+0000","last_unstale":"2024-03-26T01:58:59.179952+0000","last_undegraded":"2024-03-26T01:58:59.179952+0000","last_fullsized":"2024-03-26T01:58:59.179952+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:00:42.354013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.186104+0000","last_change":"2024-03-26T01:58:30.139735+0000","last_active":"2024-03-26T01:58:59.186104+0000","last_peered":"2024-03-26T01:58:59.186104+0000","last_clean":"2024-03-26T01:58:59.186104+0000","last_became_active":"2024-03-26T01:58:30.138545+0000","last_became_peered":"2024-03-26T01:58:30.138545+0000","last_unstale":"2024-03-26T01:58:59.186104+0000","last_undegraded":"2024-03-26T01:58:59.186104+0000","last_fullsized":"2024-03-26T01:58:59.186104+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:41:32.908074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182862+0000","last_change":"2024-03-26T01:58:32.154564+0000","last_active":"2024-03-26T01:58:59.182862+0000","last_peered":"2024-03-26T01:58:59.182862+0000","last_clean":"2024-03-26T01:58:59.182862+0000","last_became_active":"2024-03-26T01:58:32.154194+0000","last_became_peered":"2024-03-26T01:58:32.154194+0000","last_unstale":"2024-03-26T01:58:59.182862+0000","last_undegraded":"2024-03-26T01:58:59.182862+0000","last_fullsized":"2024-03-26T01:58:59.182862+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:14:57.940064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180680+0000","last_change":"2024-03-26T01:58:34.171756+0000","last_active":"2024-03-26T01:58:59.180680+0000","last_peered":"2024-03-26T01:58:59.180680+0000","last_clean":"2024-03-26T01:58:59.180680+0000","last_became_active":"2024-03-26T01:58:34.171327+0000","last_became_peered":"2024-03-26T01:58:34.171327+0000","last_unstale":"2024-03-26T01:58:59.180680+0000","last_undegraded":"2024-03-26T01:58:59.180680+0000","last_fullsized":"2024-03-26T01:58:59.180680+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:58:38.598302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.180170+0000","last_change":"2024-03-26T01:58:36.188667+0000","last_active":"2024-03-26T01:58:59.180170+0000","last_peered":"2024-03-26T01:58:59.180170+0000","last_clean":"2024-03-26T01:58:59.180170+0000","last_became_active":"2024-03-26T01:58:36.187974+0000","last_became_peered":"2024-03-26T01:58:36.187974+0000","last_unstale":"2024-03-26T01:58:59.180170+0000","last_undegraded":"2024-03-26T01:58:59.180170+0000","last_fullsized":"2024-03-26T01:58:59.180170+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:35.166955+0000","last_clean_scrub_stamp":"2024-03-26T01:58:35.166955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:58:10.007013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.182823+0000","last_change":"2024-03-26T01:58:30.137553+0000","last_active":"2024-03-26T01:58:59.182823+0000","last_peered":"2024-03-26T01:58:59.182823+0000","last_clean":"2024-03-26T01:58:59.182823+0000","last_became_active":"2024-03-26T01:58:30.137125+0000","last_became_peered":"2024-03-26T01:58:30.137125+0000","last_unstale":"2024-03-26T01:58:59.182823+0000","last_undegraded":"2024-03-26T01:58:59.182823+0000","last_fullsized":"2024-03-26T01:58:59.182823+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:29.108061+0000","last_clean_scrub_stamp":"2024-03-26T01:58:29.108061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:21:29.748266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.183184+0000","last_change":"2024-03-26T01:58:32.159537+0000","last_active":"2024-03-26T01:58:59.183184+0000","last_peered":"2024-03-26T01:58:59.183184+0000","last_clean":"2024-03-26T01:58:59.183184+0000","last_became_active":"2024-03-26T01:58:32.159254+0000","last_became_peered":"2024-03-26T01:58:32.159254+0000","last_unstale":"2024-03-26T01:58:59.183184+0000","last_undegraded":"2024-03-26T01:58:59.183184+0000","last_fullsized":"2024-03-26T01:58:59.183184+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:31.123840+0000","last_clean_scrub_stamp":"2024-03-26T01:58:31.123840+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:20:35.912710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-26T01:58:59.184124+0000","last_change":"2024-03-26T01:58:34.171491+0000","last_active":"2024-03-26T01:58:59.184124+0000","last_peered":"2024-03-26T01:58:59.184124+0000","last_clean":"2024-03-26T01:58:59.184124+0000","last_became_active":"2024-03-26T01:58:34.171153+0000","last_became_peered":"2024-03-26T01:58:34.171153+0000","last_unstale":"2024-03-26T01:58:59.184124+0000","last_undegraded":"2024-03-26T01:58:59.184124+0000","last_fullsized":"2024-03-26T01:58:59.184124+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:33.145168+0000","last_clean_scrub_stamp":"2024-03-26T01:58:33.145168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:09:37.497089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":590,"ondisk_log_size":590,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411341,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738391,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T01:59:18.800 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T01:59:18.801 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-26T01:59:18.801 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-26T01:59:18.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-26T01:59:19.062 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:18 smithi003 ceph-mon[25810]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:19.062 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 01:59:19 smithi003 podman[54111]: 2024-03-26 01:59:19.003962269 +0000 UTC m=+0.100237865 volume create 682a33a4acfa52aea6ee1496f2bdeebdbf5b2c1ee788b2f8a08371008daf08fc 2024-03-26T01:59:19.062 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 01:59:19 smithi003 podman[54111]: 2024-03-26 01:59:18.921633205 +0000 UTC m=+0.017908780 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-26T01:59:19.062 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 01:59:19 smithi003 podman[54111]: 2024-03-26 01:59:19.062553513 +0000 UTC m=+0.158829057 container create 808d9b2a0f555b45bc96824f2452340443c8435ee772927fa8e9315100a44cf2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T01:59:19.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:18 smithi066 ceph-mon[30259]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:19.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:18 smithi055 ceph-mon[30619]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:19.381 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 01:59:19 smithi003 podman[54111]: 2024-03-26 01:59:19.382052217 +0000 UTC m=+0.478327762 container init 808d9b2a0f555b45bc96824f2452340443c8435ee772927fa8e9315100a44cf2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T01:59:19.587 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:59:19.641 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 01:59:19 smithi003 podman[54111]: 2024-03-26 01:59:19.387327758 +0000 UTC m=+0.483603302 container start 808d9b2a0f555b45bc96824f2452340443c8435ee772927fa8e9315100a44cf2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T01:59:19.641 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-alertmanager-a[54138]: ts=2024-03-26T01:59:19.399Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T01:59:19.641 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-alertmanager-a[54138]: ts=2024-03-26T01:59:19.399Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T01:59:19.641 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-alertmanager-a[54138]: ts=2024-03-26T01:59:19.400Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.3 port=9094 2024-03-26T01:59:19.641 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-alertmanager-a[54138]: ts=2024-03-26T01:59:19.401Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T01:59:19.641 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-alertmanager-a[54138]: ts=2024-03-26T01:59:19.424Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T01:59:19.641 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-alertmanager-a[54138]: ts=2024-03-26T01:59:19.425Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T01:59:19.641 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-alertmanager-a[54138]: ts=2024-03-26T01:59:19.428Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T01:59:19.641 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-alertmanager-a[54138]: ts=2024-03-26T01:59:19.428Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T01:59:19.641 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 01:59:19 smithi003 bash[54111]: 808d9b2a0f555b45bc96824f2452340443c8435ee772927fa8e9315100a44cf2 2024-03-26T01:59:19.642 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 01:59:19 smithi003 systemd[1]: Started Ceph alertmanager.a for b95eae18-eb13-11ee-95d0-87774f69a715. 2024-03-26T01:59:20.034 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-mon[25810]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:59:20.035 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:20.035 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:20.035 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:20.035 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:20.035 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:20.035 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:20.035 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T01:59:20.035 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:20.035 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: [26/Mar/2024:01:59:19] ENGINE Bus STOPPING 2024-03-26T01:59:20.035 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: [26/Mar/2024:01:59:19] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T01:59:20.036 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: [26/Mar/2024:01:59:19] ENGINE Bus STOPPED 2024-03-26T01:59:20.036 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: [26/Mar/2024:01:59:19] ENGINE Bus STARTING 2024-03-26T01:59:20.036 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: [26/Mar/2024:01:59:19] ENGINE Serving on http://:::9283 2024-03-26T01:59:20.036 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:19 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: [26/Mar/2024:01:59:19] ENGINE Bus STARTED 2024-03-26T01:59:20.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:19 smithi066 ceph-mon[30259]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:59:20.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:19 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:20.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:19 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:20.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:19 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:20.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:19 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:20.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:19 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:20.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:19 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:20.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:19 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T01:59:20.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:19 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:20.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:19 smithi055 ceph-mon[30619]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:59:20.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:19 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:20.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:19 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:20.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:19 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:20.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:19 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:20.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:19 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:20.271 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:19 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:20.271 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:19 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T01:59:20.271 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:19 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:21.015 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:20 smithi003 ceph-mon[25810]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:21.015 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:20 smithi003 ceph-mon[25810]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T01:59:21.015 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:20 smithi003 ceph-mon[25810]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T01:59:21.015 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:20 smithi003 ceph-mon[25810]: Deploying daemon grafana.a on smithi055 2024-03-26T01:59:21.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:20 smithi066 ceph-mon[30259]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:21.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:20 smithi066 ceph-mon[30259]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T01:59:21.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:20 smithi066 ceph-mon[30259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T01:59:21.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:20 smithi066 ceph-mon[30259]: Deploying daemon grafana.a on smithi055 2024-03-26T01:59:21.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:20 smithi055 ceph-mon[30619]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:21.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:20 smithi055 ceph-mon[30619]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T01:59:21.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:20 smithi055 ceph-mon[30619]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T01:59:21.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:20 smithi055 ceph-mon[30619]: Deploying daemon grafana.a on smithi055 2024-03-26T01:59:21.521 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-26T01:59:21.521 INFO:teuthology.orchestra.run.smithi003.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-26T01:59:21.777 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 01:59:21 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-alertmanager-a[54138]: ts=2024-03-26T01:59:21.402Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00056983s 2024-03-26T01:59:22.036 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:21 smithi003 ceph-mon[25810]: from='client.? 172.21.15.3:0/3532937322' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T01:59:22.037 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-26T01:59:22.038 INFO:tasks.cephadm:Setup complete, yielding 2024-03-26T01:59:22.038 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T01:59:22.050 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi003.front.sepia.ceph.com 2024-03-26T01:59:22.051 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-26T01:59:22.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:21 smithi066 ceph-mon[30259]: from='client.? 172.21.15.3:0/3532937322' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T01:59:22.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:21 smithi055 ceph-mon[30619]: from='client.? 172.21.15.3:0/3532937322' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T01:59:22.297 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:59:23.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:22 smithi003 ceph-mon[25810]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:23.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:22 smithi066 ceph-mon[30259]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:23.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:22 smithi055 ceph-mon[30619]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:24.120 INFO:teuthology.orchestra.run.smithi003.stdout:167 167 2024-03-26T01:59:24.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- bash -c 'ceph orch status' 2024-03-26T01:59:24.943 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:59:25.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:24 smithi003 ceph-mon[25810]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:25.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:24 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:25.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:24 smithi066 ceph-mon[30259]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:25.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:24 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:25.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:24 smithi055 ceph-mon[30619]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:25.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:24 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' 2024-03-26T01:59:26.733 INFO:teuthology.orchestra.run.smithi003.stdout:Backend: cephadm 2024-03-26T01:59:26.733 INFO:teuthology.orchestra.run.smithi003.stdout:Available: Yes 2024-03-26T01:59:26.733 INFO:teuthology.orchestra.run.smithi003.stdout:Paused: No 2024-03-26T01:59:27.043 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:26 smithi003 ceph-mon[25810]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:27.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:26 smithi066 ceph-mon[30259]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:27.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:26 smithi055 ceph-mon[30619]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:27.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-26T01:59:27.531 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:59:27.904 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:27 smithi003 ceph-mon[25810]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:27.904 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:27 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: ::ffff:172.21.15.55 - - [26/Mar/2024:01:59:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T01:59:28.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:27 smithi066 ceph-mon[30259]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:28.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:27 smithi055 ceph-mon[30619]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:29.128 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:28 smithi066 ceph-mon[30259]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:29.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:28 smithi003 ceph-mon[25810]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:29.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:28 smithi055 ceph-mon[30619]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:29.302 INFO:teuthology.orchestra.run.smithi003.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T01:59:29.302 INFO:teuthology.orchestra.run.smithi003.stdout:alertmanager.a smithi003 *:9093,9094 starting - - - - 2024-03-26T01:59:29.302 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.x smithi055 *:8443,8765 running (2m) 27s ago 2m 447M - 19.0.0-1775-g5e70124b 5f21c9264136 40a447c1012e 2024-03-26T01:59:29.302 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.y smithi003 *:9283,8765 running (4m) 24s ago 4m 477M - 19.0.0-1775-g5e70124b 5f21c9264136 e194b78ab17b 2024-03-26T01:59:29.302 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.z smithi066 *:8443,8765 running (2m) 28s ago 2m 448M - 19.0.0-1775-g5e70124b 5f21c9264136 c8a4bef62a3e 2024-03-26T01:59:29.302 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a smithi003 running (4m) 24s ago 4m 49.9M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 8ed421558ef9 2024-03-26T01:59:29.302 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b smithi055 running (2m) 27s ago 2m 41.9M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 6085b4c176b3 2024-03-26T01:59:29.302 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c smithi066 running (2m) 28s ago 2m 42.4M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 8b36477ce6de 2024-03-26T01:59:29.303 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 smithi003 running (2m) 24s ago 2m 52.8M 15.6G 19.0.0-1775-g5e70124b 5f21c9264136 955576f2ad5d 2024-03-26T01:59:29.303 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 smithi055 running (104s) 27s ago 103s 61.0M 15.6G 19.0.0-1775-g5e70124b 5f21c9264136 46926b4403bf 2024-03-26T01:59:29.303 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 smithi066 running (79s) 28s ago 79s 58.9M 16.6G 19.0.0-1775-g5e70124b 5f21c9264136 9a6d5e087cb2 2024-03-26T01:59:29.303 INFO:teuthology.orchestra.run.smithi003.stdout:prometheus.a smithi055 *:9095 running (38s) 27s ago 38s 24.4M - 2.43.0 a07b618ecd1d d99a45fedd93 2024-03-26T01:59:29.303 INFO:teuthology.orchestra.run.smithi003.stdout:rgw.foo.a smithi003 *:80 running (61s) 24s ago 60s 80.0M - 19.0.0-1775-g5e70124b 5f21c9264136 961bfb838631 2024-03-26T01:59:29.596 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 01:59:29 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-alertmanager-a[54138]: ts=2024-03-26T01:59:29.404Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.00242911s 2024-03-26T01:59:29.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-26T01:59:30.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:29 smithi003 ceph-mon[25810]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:59:30.141 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:59:30.145 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:29 smithi066 ceph-mon[30259]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:59:30.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:29 smithi055 ceph-mon[30619]: from='mgr.14439 172.21.15.3:0/732882009' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:59:31.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:30 smithi003 ceph-mon[25810]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:31.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:30 smithi003 ceph-mon[25810]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:31.171 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:30 smithi055 ceph-mon[30619]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:31.171 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:30 smithi055 ceph-mon[30619]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:31.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:30 smithi066 ceph-mon[30259]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:31.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:30 smithi066 ceph-mon[30259]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:31.904 INFO:teuthology.orchestra.run.smithi003.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-26T01:59:31.904 INFO:teuthology.orchestra.run.smithi003.stdout:alertmanager ?:9093,9094 0/1 - 58s smithi003=a;count:1 2024-03-26T01:59:31.904 INFO:teuthology.orchestra.run.smithi003.stdout:grafana ?:3000 0/1 - 55s smithi055=a;count:1 2024-03-26T01:59:31.904 INFO:teuthology.orchestra.run.smithi003.stdout:mgr 3/3 30s ago 2m smithi003=y;smithi055=x;smithi066=z;count:3 2024-03-26T01:59:31.904 INFO:teuthology.orchestra.run.smithi003.stdout:mon 3/3 30s ago 3m smithi003:172.21.15.3=a;smithi055:172.21.15.55=b;smithi066:172.21.15.66=c;count:3 2024-03-26T01:59:31.904 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter ?:9100 0/3 - 61s smithi003=a;smithi055=b;smithi066=c;count:3 2024-03-26T01:59:31.904 INFO:teuthology.orchestra.run.smithi003.stdout:osd 3 30s ago - 2024-03-26T01:59:31.904 INFO:teuthology.orchestra.run.smithi003.stdout:prometheus ?:9095 1/1 30s ago 63s smithi055=a;count:1 2024-03-26T01:59:31.905 INFO:teuthology.orchestra.run.smithi003.stdout:rgw.foo.a ?:80 1/1 27s ago 63s smithi003=foo.a;count:1 2024-03-26T01:59:32.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-26T01:59:32.842 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:59:33.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:32 smithi003 ceph-mon[25810]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:33.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:32 smithi003 ceph-mon[25810]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:33.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:32 smithi066 ceph-mon[30259]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:33.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:32 smithi066 ceph-mon[30259]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:33.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:32 smithi055 ceph-mon[30619]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:33.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:32 smithi055 ceph-mon[30619]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:34.616 INFO:teuthology.orchestra.run.smithi003.stdout:HOST ADDR LABELS STATUS 2024-03-26T01:59:34.617 INFO:teuthology.orchestra.run.smithi003.stdout:smithi003 172.21.15.3 2024-03-26T01:59:34.617 INFO:teuthology.orchestra.run.smithi003.stdout:smithi055 172.21.15.55 2024-03-26T01:59:34.617 INFO:teuthology.orchestra.run.smithi003.stdout:smithi066 172.21.15.66 2024-03-26T01:59:34.617 INFO:teuthology.orchestra.run.smithi003.stdout:3 hosts in cluster 2024-03-26T01:59:35.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:34 smithi003 ceph-mon[25810]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:35.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-26T01:59:35.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:34 smithi066 ceph-mon[30259]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:35.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:34 smithi055 ceph-mon[30619]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:35.420 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:59:36.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:35 smithi003 ceph-mon[25810]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:36.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:35 smithi066 ceph-mon[30259]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:36.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:35 smithi055 ceph-mon[30619]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:37.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:36 smithi066 ceph-mon[30259]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:37.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:36 smithi055 ceph-mon[30619]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:37.358 INFO:teuthology.orchestra.run.smithi003.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-26T01:59:37.358 INFO:teuthology.orchestra.run.smithi003.stdout:smithi003 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330009Z400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-26T01:59:37.358 INFO:teuthology.orchestra.run.smithi003.stdout:smithi003 /dev/nvme1n1 ssd Linux_0a0d385bc5bfdb616bfc 89.4G Yes 2m ago 2024-03-26T01:59:37.358 INFO:teuthology.orchestra.run.smithi003.stdout:smithi003 /dev/nvme2n1 ssd Linux_74094c002ec1a6356a99 89.4G Yes 2m ago 2024-03-26T01:59:37.359 INFO:teuthology.orchestra.run.smithi003.stdout:smithi003 /dev/nvme3n1 ssd Linux_63e9e6c6c5772d9eb7c8 89.4G Yes 2m ago 2024-03-26T01:59:37.359 INFO:teuthology.orchestra.run.smithi003.stdout:smithi003 /dev/nvme4n1 ssd Linux_289d483ded2146e17ecb 89.4G No 2m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-26T01:59:37.359 INFO:teuthology.orchestra.run.smithi003.stdout:smithi055 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736102VS4P0KGN 3726G No 104s ago Has a FileSystem, LVM detected 2024-03-26T01:59:37.359 INFO:teuthology.orchestra.run.smithi003.stdout:smithi055 /dev/nvme1n1 ssd Linux_946394b4ad2caf1bfbcc 89.4G Yes 104s ago 2024-03-26T01:59:37.359 INFO:teuthology.orchestra.run.smithi003.stdout:smithi055 /dev/nvme2n1 ssd Linux_56050468d35d5895d437 89.4G Yes 104s ago 2024-03-26T01:59:37.359 INFO:teuthology.orchestra.run.smithi003.stdout:smithi055 /dev/nvme3n1 ssd Linux_536c0db306eadca59576 89.4G Yes 104s ago 2024-03-26T01:59:37.359 INFO:teuthology.orchestra.run.smithi003.stdout:smithi055 /dev/nvme4n1 ssd Linux_5ec00d0f00f1bfc54eba 89.4G No 104s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-26T01:59:37.359 INFO:teuthology.orchestra.run.smithi003.stdout:smithi066 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53430091400BGN 372G No 79s ago Has a FileSystem, LVM detected 2024-03-26T01:59:37.359 INFO:teuthology.orchestra.run.smithi003.stdout:smithi066 /dev/nvme1n1 ssd Linux_8a4618dc01cba413d408 89.4G Yes 79s ago 2024-03-26T01:59:37.359 INFO:teuthology.orchestra.run.smithi003.stdout:smithi066 /dev/nvme2n1 ssd Linux_bb1711b6ce20a44bf57c 89.4G Yes 79s ago 2024-03-26T01:59:37.359 INFO:teuthology.orchestra.run.smithi003.stdout:smithi066 /dev/nvme3n1 ssd Linux_56c7c153a120656585f3 89.4G Yes 79s ago 2024-03-26T01:59:37.359 INFO:teuthology.orchestra.run.smithi003.stdout:smithi066 /dev/nvme4n1 ssd Linux_6f879da8a4139b8483be 89.4G No 79s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-26T01:59:37.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:36 smithi003 ceph-mon[25810]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:37.927 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:37 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: ::ffff:172.21.15.55 - - [26/Mar/2024:01:59:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T01:59:37.954 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-26T01:59:38.195 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:59:39.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:38 smithi066 ceph-mon[30259]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:39.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:38 smithi066 ceph-mon[30259]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:39.266 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:38 smithi003 ceph-mon[25810]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:39.267 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:38 smithi003 ceph-mon[25810]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:39.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:38 smithi055 ceph-mon[30619]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:39.270 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:38 smithi055 ceph-mon[30619]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:40.007 INFO:teuthology.orchestra.run.smithi003.stdout:service_type: alertmanager 2024-03-26T01:59:40.007 INFO:teuthology.orchestra.run.smithi003.stdout:service_name: alertmanager 2024-03-26T01:59:40.007 INFO:teuthology.orchestra.run.smithi003.stdout:placement: 2024-03-26T01:59:40.007 INFO:teuthology.orchestra.run.smithi003.stdout: count: 1 2024-03-26T01:59:40.007 INFO:teuthology.orchestra.run.smithi003.stdout: hosts: 2024-03-26T01:59:40.007 INFO:teuthology.orchestra.run.smithi003.stdout: - smithi003=a 2024-03-26T01:59:40.007 INFO:teuthology.orchestra.run.smithi003.stdout:status: 2024-03-26T01:59:40.007 INFO:teuthology.orchestra.run.smithi003.stdout: created: '2024-03-26T01:58:33.722434Z' 2024-03-26T01:59:40.007 INFO:teuthology.orchestra.run.smithi003.stdout: ports: 2024-03-26T01:59:40.007 INFO:teuthology.orchestra.run.smithi003.stdout: - 9093 2024-03-26T01:59:40.007 INFO:teuthology.orchestra.run.smithi003.stdout: - 9094 2024-03-26T01:59:40.008 INFO:teuthology.orchestra.run.smithi003.stdout: running: 0 2024-03-26T01:59:40.008 INFO:teuthology.orchestra.run.smithi003.stdout: size: 1 2024-03-26T01:59:40.008 INFO:teuthology.orchestra.run.smithi003.stdout:events: 2024-03-26T01:59:40.008 INFO:teuthology.orchestra.run.smithi003.stdout:- 2024-03-26T01:59:19.587043Z service:alertmanager [INFO] "service was created" 2024-03-26T01:59:40.008 INFO:teuthology.orchestra.run.smithi003.stdout:--- 2024-03-26T01:59:40.008 INFO:teuthology.orchestra.run.smithi003.stdout:service_type: grafana 2024-03-26T01:59:40.008 INFO:teuthology.orchestra.run.smithi003.stdout:service_name: grafana 2024-03-26T01:59:40.008 INFO:teuthology.orchestra.run.smithi003.stdout:placement: 2024-03-26T01:59:40.008 INFO:teuthology.orchestra.run.smithi003.stdout: count: 1 2024-03-26T01:59:40.008 INFO:teuthology.orchestra.run.smithi003.stdout: hosts: 2024-03-26T01:59:40.008 INFO:teuthology.orchestra.run.smithi003.stdout: - smithi055=a 2024-03-26T01:59:40.008 INFO:teuthology.orchestra.run.smithi003.stdout:spec: 2024-03-26T01:59:40.008 INFO:teuthology.orchestra.run.smithi003.stdout: anonymous_access: true 2024-03-26T01:59:40.008 INFO:teuthology.orchestra.run.smithi003.stdout: protocol: https 2024-03-26T01:59:40.009 INFO:teuthology.orchestra.run.smithi003.stdout:status: 2024-03-26T01:59:40.009 INFO:teuthology.orchestra.run.smithi003.stdout: created: '2024-03-26T01:58:36.518190Z' 2024-03-26T01:59:40.009 INFO:teuthology.orchestra.run.smithi003.stdout: ports: 2024-03-26T01:59:40.009 INFO:teuthology.orchestra.run.smithi003.stdout: - 3000 2024-03-26T01:59:40.009 INFO:teuthology.orchestra.run.smithi003.stdout: running: 0 2024-03-26T01:59:40.009 INFO:teuthology.orchestra.run.smithi003.stdout: size: 1 2024-03-26T01:59:40.009 INFO:teuthology.orchestra.run.smithi003.stdout:--- 2024-03-26T01:59:40.009 INFO:teuthology.orchestra.run.smithi003.stdout:service_type: mgr 2024-03-26T01:59:40.009 INFO:teuthology.orchestra.run.smithi003.stdout:service_name: mgr 2024-03-26T01:59:40.009 INFO:teuthology.orchestra.run.smithi003.stdout:placement: 2024-03-26T01:59:40.009 INFO:teuthology.orchestra.run.smithi003.stdout: count: 3 2024-03-26T01:59:40.009 INFO:teuthology.orchestra.run.smithi003.stdout: hosts: 2024-03-26T01:59:40.009 INFO:teuthology.orchestra.run.smithi003.stdout: - smithi003=y 2024-03-26T01:59:40.009 INFO:teuthology.orchestra.run.smithi003.stdout: - smithi055=x 2024-03-26T01:59:40.009 INFO:teuthology.orchestra.run.smithi003.stdout: - smithi066=z 2024-03-26T01:59:40.010 INFO:teuthology.orchestra.run.smithi003.stdout:status: 2024-03-26T01:59:40.010 INFO:teuthology.orchestra.run.smithi003.stdout: created: '2024-03-26T01:56:56.592414Z' 2024-03-26T01:59:40.010 INFO:teuthology.orchestra.run.smithi003.stdout: last_refresh: '2024-03-26T01:59:00.970290Z' 2024-03-26T01:59:40.010 INFO:teuthology.orchestra.run.smithi003.stdout: running: 3 2024-03-26T01:59:40.010 INFO:teuthology.orchestra.run.smithi003.stdout: size: 3 2024-03-26T01:59:40.010 INFO:teuthology.orchestra.run.smithi003.stdout:--- 2024-03-26T01:59:40.010 INFO:teuthology.orchestra.run.smithi003.stdout:service_type: mon 2024-03-26T01:59:40.010 INFO:teuthology.orchestra.run.smithi003.stdout:service_name: mon 2024-03-26T01:59:40.010 INFO:teuthology.orchestra.run.smithi003.stdout:placement: 2024-03-26T01:59:40.010 INFO:teuthology.orchestra.run.smithi003.stdout: count: 3 2024-03-26T01:59:40.010 INFO:teuthology.orchestra.run.smithi003.stdout: hosts: 2024-03-26T01:59:40.010 INFO:teuthology.orchestra.run.smithi003.stdout: - smithi003:172.21.15.3=a 2024-03-26T01:59:40.010 INFO:teuthology.orchestra.run.smithi003.stdout: - smithi055:172.21.15.55=b 2024-03-26T01:59:40.010 INFO:teuthology.orchestra.run.smithi003.stdout: - smithi066:172.21.15.66=c 2024-03-26T01:59:40.011 INFO:teuthology.orchestra.run.smithi003.stdout:status: 2024-03-26T01:59:40.011 INFO:teuthology.orchestra.run.smithi003.stdout: created: '2024-03-26T01:56:29.604018Z' 2024-03-26T01:59:40.011 INFO:teuthology.orchestra.run.smithi003.stdout: last_refresh: '2024-03-26T01:59:00.970086Z' 2024-03-26T01:59:40.011 INFO:teuthology.orchestra.run.smithi003.stdout: running: 3 2024-03-26T01:59:40.011 INFO:teuthology.orchestra.run.smithi003.stdout: size: 3 2024-03-26T01:59:40.011 INFO:teuthology.orchestra.run.smithi003.stdout:--- 2024-03-26T01:59:40.011 INFO:teuthology.orchestra.run.smithi003.stdout:service_type: node-exporter 2024-03-26T01:59:40.011 INFO:teuthology.orchestra.run.smithi003.stdout:service_name: node-exporter 2024-03-26T01:59:40.011 INFO:teuthology.orchestra.run.smithi003.stdout:placement: 2024-03-26T01:59:40.011 INFO:teuthology.orchestra.run.smithi003.stdout: count: 3 2024-03-26T01:59:40.011 INFO:teuthology.orchestra.run.smithi003.stdout: hosts: 2024-03-26T01:59:40.011 INFO:teuthology.orchestra.run.smithi003.stdout: - smithi003=a 2024-03-26T01:59:40.011 INFO:teuthology.orchestra.run.smithi003.stdout: - smithi055=b 2024-03-26T01:59:40.012 INFO:teuthology.orchestra.run.smithi003.stdout: - smithi066=c 2024-03-26T01:59:40.012 INFO:teuthology.orchestra.run.smithi003.stdout:status: 2024-03-26T01:59:40.012 INFO:teuthology.orchestra.run.smithi003.stdout: created: '2024-03-26T01:58:30.884659Z' 2024-03-26T01:59:40.012 INFO:teuthology.orchestra.run.smithi003.stdout: ports: 2024-03-26T01:59:40.012 INFO:teuthology.orchestra.run.smithi003.stdout: - 9100 2024-03-26T01:59:40.012 INFO:teuthology.orchestra.run.smithi003.stdout: running: 0 2024-03-26T01:59:40.012 INFO:teuthology.orchestra.run.smithi003.stdout: size: 3 2024-03-26T01:59:40.012 INFO:teuthology.orchestra.run.smithi003.stdout:--- 2024-03-26T01:59:40.012 INFO:teuthology.orchestra.run.smithi003.stdout:service_type: osd 2024-03-26T01:59:40.013 INFO:teuthology.orchestra.run.smithi003.stdout:service_name: osd 2024-03-26T01:59:40.013 INFO:teuthology.orchestra.run.smithi003.stdout:unmanaged: true 2024-03-26T01:59:40.013 INFO:teuthology.orchestra.run.smithi003.stdout:spec: 2024-03-26T01:59:40.013 INFO:teuthology.orchestra.run.smithi003.stdout: filter_logic: AND 2024-03-26T01:59:40.013 INFO:teuthology.orchestra.run.smithi003.stdout: objectstore: bluestore 2024-03-26T01:59:40.013 INFO:teuthology.orchestra.run.smithi003.stdout:status: 2024-03-26T01:59:40.013 INFO:teuthology.orchestra.run.smithi003.stdout: container_image_id: 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T01:59:40.013 INFO:teuthology.orchestra.run.smithi003.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T01:59:40.013 INFO:teuthology.orchestra.run.smithi003.stdout: last_refresh: '2024-03-26T01:59:00.970469Z' 2024-03-26T01:59:40.013 INFO:teuthology.orchestra.run.smithi003.stdout: running: 3 2024-03-26T01:59:40.013 INFO:teuthology.orchestra.run.smithi003.stdout: size: 3 2024-03-26T01:59:40.013 INFO:teuthology.orchestra.run.smithi003.stdout:--- 2024-03-26T01:59:40.013 INFO:teuthology.orchestra.run.smithi003.stdout:service_type: prometheus 2024-03-26T01:59:40.014 INFO:teuthology.orchestra.run.smithi003.stdout:service_name: prometheus 2024-03-26T01:59:40.014 INFO:teuthology.orchestra.run.smithi003.stdout:placement: 2024-03-26T01:59:40.014 INFO:teuthology.orchestra.run.smithi003.stdout: count: 1 2024-03-26T01:59:40.014 INFO:teuthology.orchestra.run.smithi003.stdout: hosts: 2024-03-26T01:59:40.014 INFO:teuthology.orchestra.run.smithi003.stdout: - smithi055=a 2024-03-26T01:59:40.014 INFO:teuthology.orchestra.run.smithi003.stdout:status: 2024-03-26T01:59:40.014 INFO:teuthology.orchestra.run.smithi003.stdout: created: '2024-03-26T01:58:28.105902Z' 2024-03-26T01:59:40.014 INFO:teuthology.orchestra.run.smithi003.stdout: last_refresh: '2024-03-26T01:59:01.687146Z' 2024-03-26T01:59:40.014 INFO:teuthology.orchestra.run.smithi003.stdout: ports: 2024-03-26T01:59:40.014 INFO:teuthology.orchestra.run.smithi003.stdout: - 9095 2024-03-26T01:59:40.014 INFO:teuthology.orchestra.run.smithi003.stdout: running: 1 2024-03-26T01:59:40.014 INFO:teuthology.orchestra.run.smithi003.stdout: size: 1 2024-03-26T01:59:40.015 INFO:teuthology.orchestra.run.smithi003.stdout:--- 2024-03-26T01:59:40.015 INFO:teuthology.orchestra.run.smithi003.stdout:service_type: rgw 2024-03-26T01:59:40.015 INFO:teuthology.orchestra.run.smithi003.stdout:service_id: foo.a 2024-03-26T01:59:40.015 INFO:teuthology.orchestra.run.smithi003.stdout:service_name: rgw.foo.a 2024-03-26T01:59:40.015 INFO:teuthology.orchestra.run.smithi003.stdout:placement: 2024-03-26T01:59:40.015 INFO:teuthology.orchestra.run.smithi003.stdout: count: 1 2024-03-26T01:59:40.015 INFO:teuthology.orchestra.run.smithi003.stdout: hosts: 2024-03-26T01:59:40.015 INFO:teuthology.orchestra.run.smithi003.stdout: - smithi003=foo.a 2024-03-26T01:59:40.015 INFO:teuthology.orchestra.run.smithi003.stdout:status: 2024-03-26T01:59:40.015 INFO:teuthology.orchestra.run.smithi003.stdout: created: '2024-03-26T01:58:28.815272Z' 2024-03-26T01:59:40.015 INFO:teuthology.orchestra.run.smithi003.stdout: last_refresh: '2024-03-26T01:59:04.393223Z' 2024-03-26T01:59:40.015 INFO:teuthology.orchestra.run.smithi003.stdout: ports: 2024-03-26T01:59:40.015 INFO:teuthology.orchestra.run.smithi003.stdout: - 80 2024-03-26T01:59:40.015 INFO:teuthology.orchestra.run.smithi003.stdout: running: 1 2024-03-26T01:59:40.015 INFO:teuthology.orchestra.run.smithi003.stdout: size: 1 2024-03-26T01:59:40.784 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b95eae18-eb13-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-26T01:59:41.029 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:59:41.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:40 smithi003 ceph-mon[25810]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:41.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:40 smithi003 ceph-mon[25810]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-26T01:59:41.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:40 smithi066 ceph-mon[30259]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:41.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:40 smithi066 ceph-mon[30259]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-26T01:59:41.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:40 smithi055 ceph-mon[30619]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:41.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:40 smithi055 ceph-mon[30619]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-26T01:59:42.878 INFO:teuthology.orchestra.run.smithi003.stdout:osd 3 41s ago - 2024-03-26T01:59:43.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:42 smithi003 ceph-mon[25810]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:43.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:42 smithi066 ceph-mon[30259]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:43.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:42 smithi055 ceph-mon[30619]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:43.481 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-26T01:59:43.491 INFO:tasks.cephadm:Teardown begin 2024-03-26T01:59:43.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T01:59:43.522 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T01:59:43.555 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T01:59:43.595 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-26T01:59:43.595 DEBUG:teuthology.orchestra.run.smithi003:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T01:59:43.611 DEBUG:teuthology.orchestra.run.smithi055:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T01:59:43.627 DEBUG:teuthology.orchestra.run.smithi066:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T01:59:43.654 INFO:tasks.cephadm:Stopping all daemons... 2024-03-26T01:59:43.655 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-26T01:59:43.655 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mon.a 2024-03-26T01:59:43.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:43 smithi003 systemd[1]: Stopping Ceph mon.a for b95eae18-eb13-11ee-95d0-87774f69a715... 2024-03-26T01:59:44.227 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:43 smithi066 ceph-mon[30259]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:44.269 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:43 smithi055 ceph-mon[30619]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:44.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:43 smithi003 ceph-mon[25810]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:44.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:43 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-a[25787]: 2024-03-26T01:59:43.943+0000 7f2c3e35a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T01:59:44.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:43 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-a[25787]: 2024-03-26T01:59:43.943+0000 7f2c3e35a700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-26T01:59:44.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:44 smithi003 podman[56109]: 2024-03-26 01:59:44.004856579 +0000 UTC m=+0.161467537 container died 8ed421558ef9f3e7ae4c71a89a2dc723a4bd61983231292a8684760900ce059d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T01:59:44.283 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:43 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: [26/Mar/2024:01:59:43] ENGINE Bus STOPPING 2024-03-26T01:59:44.283 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:43 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: [26/Mar/2024:01:59:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T01:59:44.283 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:43 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: [26/Mar/2024:01:59:43] ENGINE Bus STOPPED 2024-03-26T01:59:44.284 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:43 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: [26/Mar/2024:01:59:43] ENGINE Bus STARTING 2024-03-26T01:59:44.284 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:44 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: [26/Mar/2024:01:59:44] ENGINE Serving on http://:::9283 2024-03-26T01:59:44.284 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:44 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: [26/Mar/2024:01:59:44] ENGINE Bus STARTED 2024-03-26T01:59:44.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:44 smithi003 podman[56109]: 2024-03-26 01:59:44.473586989 +0000 UTC m=+0.630197955 container cleanup 8ed421558ef9f3e7ae4c71a89a2dc723a4bd61983231292a8684760900ce059d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124) 2024-03-26T01:59:44.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 26 01:59:44 smithi003 bash[56109]: ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-a 2024-03-26T01:59:44.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mon.a.service' 2024-03-26T01:59:45.002 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T01:59:45.002 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-26T01:59:45.003 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-26T01:59:45.003 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mon.b 2024-03-26T01:59:45.308 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:45 smithi055 systemd[1]: Stopping Ceph mon.b for b95eae18-eb13-11ee-95d0-87774f69a715... 2024-03-26T01:59:45.613 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:45 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-b[30596]: 2024-03-26T01:59:45.307+0000 7f2a56d44700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T01:59:45.613 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:45 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-b[30596]: 2024-03-26T01:59:45.307+0000 7f2a56d44700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-26T01:59:45.613 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:45 smithi055 podman[43486]: 2024-03-26 01:59:45.34948215 +0000 UTC m=+0.152467138 container died 6085b4c176b300748781c9cf6b2fced119c6f9d35fe10a0b18e83cb6b0a61e19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-26T01:59:45.640 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:45 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: [26/Mar/2024:01:59:45] ENGINE Bus STOPPING 2024-03-26T01:59:45.640 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:45 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: [26/Mar/2024:01:59:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T01:59:45.641 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:45 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: [26/Mar/2024:01:59:45] ENGINE Bus STOPPED 2024-03-26T01:59:45.641 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:45 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: [26/Mar/2024:01:59:45] ENGINE Bus STARTING 2024-03-26T01:59:45.641 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:45 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: [26/Mar/2024:01:59:45] ENGINE Serving on http://:::9283 2024-03-26T01:59:45.641 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:45 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: [26/Mar/2024:01:59:45] ENGINE Bus STARTED 2024-03-26T01:59:45.955 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:45 smithi055 podman[43486]: 2024-03-26 01:59:45.779952841 +0000 UTC m=+0.582937878 container cleanup 6085b4c176b300748781c9cf6b2fced119c6f9d35fe10a0b18e83cb6b0a61e19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-26T01:59:45.956 INFO:journalctl@ceph.mon.b.smithi055.stdout:Mar 26 01:59:45 smithi055 bash[43486]: ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-b 2024-03-26T01:59:46.176 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mon.b.service' 2024-03-26T01:59:46.218 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T01:59:46.219 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-26T01:59:46.219 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-26T01:59:46.219 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mon.c 2024-03-26T01:59:46.534 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:46 smithi066 systemd[1]: Stopping Ceph mon.c for b95eae18-eb13-11ee-95d0-87774f69a715... 2024-03-26T01:59:46.789 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:46 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-c[30209]: 2024-03-26T01:59:46.532+0000 7fede058a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T01:59:46.789 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:46 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-c[30209]: 2024-03-26T01:59:46.532+0000 7fede058a700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-26T01:59:47.199 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:46 smithi066 podman[44014]: 2024-03-26 01:59:46.789786302 +0000 UTC m=+0.361719565 container died 8b36477ce6de8e88839efb4eeede4f153992258ae0d41f554a7589c2b0f92b5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-c, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-26T01:59:47.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:47 smithi066 podman[44014]: 2024-03-26 01:59:47.364117873 +0000 UTC m=+0.936051136 container cleanup 8b36477ce6de8e88839efb4eeede4f153992258ae0d41f554a7589c2b0f92b5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2024-03-26T01:59:47.478 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 01:59:47 smithi066 bash[44014]: ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mon-c 2024-03-26T01:59:47.736 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mon.c.service' 2024-03-26T01:59:47.783 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T01:59:47.783 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-26T01:59:47.783 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-26T01:59:47.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mgr.y 2024-03-26T01:59:48.130 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:47 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y[26065]: ::ffff:172.21.15.55 - - [26/Mar/2024:01:59:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T01:59:48.130 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:47 smithi003 systemd[1]: Stopping Ceph mgr.y for b95eae18-eb13-11ee-95d0-87774f69a715... 2024-03-26T01:59:48.390 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:48 smithi003 podman[56262]: 2024-03-26 01:59:48.136083471 +0000 UTC m=+0.150090675 container died e194b78ab17bbd40410c7f64566f054656f2c15a48a72200925a93a073b5e9ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-26T01:59:48.846 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:48 smithi003 podman[56262]: 2024-03-26 01:59:48.519425002 +0000 UTC m=+0.533432244 container cleanup e194b78ab17bbd40410c7f64566f054656f2c15a48a72200925a93a073b5e9ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T01:59:48.846 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:48 smithi003 bash[56262]: ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y 2024-03-26T01:59:48.846 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 26 01:59:48 smithi003 podman[56281]: 2024-03-26 01:59:48.737269706 +0000 UTC m=+0.600009330 container remove e194b78ab17bbd40410c7f64566f054656f2c15a48a72200925a93a073b5e9ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-y, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T01:59:48.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-26T01:59:48.895 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T01:59:48.895 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-26T01:59:48.896 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-26T01:59:48.896 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mgr.x 2024-03-26T01:59:49.232 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:59:48 smithi055 systemd[1]: Stopping Ceph mgr.x for b95eae18-eb13-11ee-95d0-87774f69a715... 2024-03-26T01:59:49.520 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:59:49 smithi055 podman[43618]: 2024-03-26 01:59:49.23292517 +0000 UTC m=+0.140808815 container died 40a447c1012e483f9c6cee59dbf08ed6474ffaed96eb9a64c4ec372e2f64a74d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS) 2024-03-26T01:59:49.886 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:59:49 smithi055 podman[43618]: 2024-03-26 01:59:49.626212329 +0000 UTC m=+0.534095982 container cleanup 40a447c1012e483f9c6cee59dbf08ed6474ffaed96eb9a64c4ec372e2f64a74d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-26T01:59:49.886 INFO:journalctl@ceph.mgr.x.smithi055.stdout:Mar 26 01:59:49 smithi055 bash[43618]: ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-x 2024-03-26T01:59:50.052 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-26T01:59:50.090 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T01:59:50.091 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-26T01:59:50.091 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-26T01:59:50.091 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mgr.z 2024-03-26T01:59:50.409 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:59:50 smithi066 systemd[1]: Stopping Ceph mgr.z for b95eae18-eb13-11ee-95d0-87774f69a715... 2024-03-26T01:59:50.725 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:59:50 smithi066 podman[44147]: 2024-03-26 01:59:50.409249304 +0000 UTC m=+0.119505285 container died c8a4bef62a3e5eba405f5f8414adbb002a10b32135e00d306a96a1b6e90c914c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-26T01:59:50.725 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:59:50 smithi066 podman[44147]: 2024-03-26 01:59:50.725799869 +0000 UTC m=+0.436055850 container cleanup c8a4bef62a3e5eba405f5f8414adbb002a10b32135e00d306a96a1b6e90c914c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T01:59:50.977 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:59:50 smithi066 bash[44147]: ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z 2024-03-26T01:59:50.977 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:59:50 smithi066 podman[44164]: 2024-03-26 01:59:50.943587457 +0000 UTC m=+0.529598333 container remove c8a4bef62a3e5eba405f5f8414adbb002a10b32135e00d306a96a1b6e90c914c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-26T01:59:50.977 INFO:journalctl@ceph.mgr.z.smithi066.stdout:Mar 26 01:59:50 smithi066 systemd[1]: ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-03-26T01:59:51.074 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@mgr.z.service' 2024-03-26T01:59:51.113 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T01:59:51.114 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-26T01:59:51.114 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-26T01:59:51.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-b95eae18-eb13-11ee-95d0-87774f69a715@osd.0 2024-03-26T01:59:51.269 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:59:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:59:50.834Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.3:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.3:8765: connect: connection refused" 2024-03-26T01:59:51.269 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 01:59:50 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T01:59:50.834Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.3:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.3:8765: connect: connection refused" 2024-03-26T01:59:51.467 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:59:51 smithi003 systemd[1]: Stopping Ceph osd.0 for b95eae18-eb13-11ee-95d0-87774f69a715... 2024-03-26T01:59:51.890 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:59:51 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-0[40104]: 2024-03-26T01:59:51.467+0000 7f6042fb2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T01:59:51.891 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:59:51 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-0[40104]: 2024-03-26T01:59:51.467+0000 7f6042fb2700 -1 osd.0 32 *** Got signal Terminated *** 2024-03-26T01:59:51.891 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:59:51 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-0[40104]: 2024-03-26T01:59:51.467+0000 7f6042fb2700 -1 osd.0 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T01:59:57.141 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:59:56 smithi003 podman[56392]: 2024-03-26 01:59:56.884016998 +0000 UTC m=+5.578576288 container died 955576f2ad5dc49f7f3891c2f4ff134ae816227ed20dd62ee0195efefb9267e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T01:59:57.473 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:59:57 smithi003 podman[56392]: 2024-03-26 01:59:57.325724856 +0000 UTC m=+6.020284185 container cleanup 955576f2ad5dc49f7f3891c2f4ff134ae816227ed20dd62ee0195efefb9267e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-0, RELEASE=squid-5e70124, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-26T01:59:57.473 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:59:57 smithi003 bash[56392]: ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-0 2024-03-26T01:59:57.890 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:59:57 smithi003 podman[56414]: 2024-03-26 01:59:57.473602637 +0000 UTC m=+0.590844471 container remove 955576f2ad5dc49f7f3891c2f4ff134ae816227ed20dd62ee0195efefb9267e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-5e70124, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-26T01:59:58.141 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:59:57 smithi003 podman[56513]: 2024-03-26 01:59:57.793541372 +0000 UTC m=+0.017670129 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T01:59:58.141 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:59:57 smithi003 podman[56513]: 2024-03-26 01:59:57.890885074 +0000 UTC m=+0.115013814 container create 1647cd1ed0efd9a25e4886b942dc1f214aa9c69db48a7123104fbda45517e881 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-0-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-26T01:59:58.141 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:59:58 smithi003 podman[56513]: 2024-03-26 01:59:58.085094321 +0000 UTC m=+0.309223062 container init 1647cd1ed0efd9a25e4886b942dc1f214aa9c69db48a7123104fbda45517e881 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T01:59:58.141 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:59:58 smithi003 podman[56513]: 2024-03-26 01:59:58.090506983 +0000 UTC m=+0.314635726 container start 1647cd1ed0efd9a25e4886b942dc1f214aa9c69db48a7123104fbda45517e881 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T01:59:58.402 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:59:58 smithi003 podman[56513]: 2024-03-26 01:59:58.184863173 +0000 UTC m=+0.408991924 container attach 1647cd1ed0efd9a25e4886b942dc1f214aa9c69db48a7123104fbda45517e881 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-26T01:59:58.845 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 26 01:59:58 smithi003 podman[56513]: 2024-03-26 01:59:58.403322772 +0000 UTC m=+0.627451514 container died 1647cd1ed0efd9a25e4886b942dc1f214aa9c69db48a7123104fbda45517e881 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-26T01:59:59.137 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@osd.0.service' 2024-03-26T01:59:59.183 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T01:59:59.184 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-26T01:59:59.184 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-26T01:59:59.184 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-b95eae18-eb13-11ee-95d0-87774f69a715@osd.1 2024-03-26T01:59:59.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 01:59:59 smithi055 systemd[1]: Stopping Ceph osd.1 for b95eae18-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:00:00.269 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 01:59:59 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-1[35177]: 2024-03-26T01:59:59.951+0000 7f20b709c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:00:00.270 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 01:59:59 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-1[35177]: 2024-03-26T01:59:59.951+0000 7f20b709c700 -1 osd.1 32 *** Got signal Terminated *** 2024-03-26T02:00:00.270 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 01:59:59 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-1[35177]: 2024-03-26T01:59:59.951+0000 7f20b709c700 -1 osd.1 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:00:03.520 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 podman[44006]: 2024-03-26 02:00:03.492740562 +0000 UTC m=+0.328525626 container start 61c884ae3291fefb1cae93917d7af8050da08b4fd7314484cfcd0407d025bb57 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a, release=239.1651231664, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, com.redhat.component=ubi8-container, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, description=Ceph Grafana Container, io.buildah.version=1.32.0, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, vendor=Red Hat, Inc.) 2024-03-26T02:00:03.810 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 bash[44006]: 61c884ae3291fefb1cae93917d7af8050da08b4fd7314484cfcd0407d025bb57 2024-03-26T02:00:03.810 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 systemd[1]: Started Ceph grafana.a for b95eae18-eb13-11ee-95d0-87774f69a715. 2024-03-26T02:00:03.810 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=settings t=2024-03-26T02:00:03.659747932Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-26T02:00:03.810 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=settings t=2024-03-26T02:00:03.659943163Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-26T02:00:03.810 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=settings t=2024-03-26T02:00:03.659962709Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-26T02:00:03.810 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=settings t=2024-03-26T02:00:03.659970114Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-26T02:00:03.811 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=settings t=2024-03-26T02:00:03.659976852Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-26T02:00:03.811 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=settings t=2024-03-26T02:00:03.659983272Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-26T02:00:03.811 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=settings t=2024-03-26T02:00:03.659990129Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-26T02:00:03.811 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=settings t=2024-03-26T02:00:03.659996696Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-26T02:00:03.811 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=settings t=2024-03-26T02:00:03.660003175Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-26T02:00:03.811 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=settings t=2024-03-26T02:00:03.660009408Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-26T02:00:03.811 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=settings t=2024-03-26T02:00:03.660015415Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-26T02:00:03.811 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=settings t=2024-03-26T02:00:03.660022937Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-26T02:00:03.811 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=settings t=2024-03-26T02:00:03.660032945Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-26T02:00:03.811 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=settings t=2024-03-26T02:00:03.660039212Z level=info msg="App mode production" 2024-03-26T02:00:03.811 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=sqlstore t=2024-03-26T02:00:03.660082964Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-26T02:00:03.811 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=sqlstore t=2024-03-26T02:00:03.66010002Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-26T02:00:03.811 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.660917926Z level=info msg="Starting DB migrations" 2024-03-26T02:00:03.811 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.660999186Z level=info msg="Executing migration" id="create migration_log table" 2024-03-26T02:00:03.813 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.661700304Z level=info msg="Executing migration" id="create user table" 2024-03-26T02:00:03.813 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.663688329Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-26T02:00:03.813 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.664400435Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-26T02:00:03.813 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.666351444Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-26T02:00:03.813 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.666997783Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-26T02:00:03.813 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.667619193Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-26T02:00:03.813 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.669510929Z level=info msg="Executing migration" id="create user table v2" 2024-03-26T02:00:03.813 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.671486934Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-26T02:00:03.813 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.672005362Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-26T02:00:03.814 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.672671042Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-26T02:00:03.814 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.674528416Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-26T02:00:03.814 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.676478566Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-26T02:00:03.814 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.677008306Z level=info msg="Executing migration" id="Update user table charset" 2024-03-26T02:00:03.814 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.677413882Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-26T02:00:03.814 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.677893779Z level=info msg="Executing migration" id="Add missing user data" 2024-03-26T02:00:03.814 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.678351983Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-26T02:00:03.814 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.678848916Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-26T02:00:03.814 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.679358352Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-26T02:00:03.814 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.679845269Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-26T02:00:03.814 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.68107823Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-26T02:00:03.814 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.681605376Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-26T02:00:03.814 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.68214874Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-26T02:00:03.815 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.682651146Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-26T02:00:03.815 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.683187132Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-26T02:00:03.815 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.683696856Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-26T02:00:03.815 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.684107667Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-26T02:00:03.815 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.684599127Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-26T02:00:03.815 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.685119866Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-26T02:00:03.815 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.685610997Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-26T02:00:03.815 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.686120206Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-26T02:00:03.815 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.686869538Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-26T02:00:03.815 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.687367619Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-26T02:00:03.815 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.687852065Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-26T02:00:03.815 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.688385569Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-26T02:00:03.815 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.688862498Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-26T02:00:03.816 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.689388597Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-26T02:00:03.816 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.689797145Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-26T02:00:03.816 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.690262204Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-26T02:00:03.816 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.690673445Z level=info msg="Executing migration" id="create star table" 2024-03-26T02:00:03.816 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.691156983Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-26T02:00:03.816 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.691691925Z level=info msg="Executing migration" id="create org table v1" 2024-03-26T02:00:03.816 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.692209688Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-26T02:00:03.816 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.692725195Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-26T02:00:03.816 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.693227949Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-26T02:00:03.816 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.693770761Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-26T02:00:03.816 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.69428843Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-26T02:00:03.816 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.694784152Z level=info msg="Executing migration" id="Update org table charset" 2024-03-26T02:00:03.816 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.695183895Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-26T02:00:03.816 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.695555802Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-26T02:00:03.817 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.69594695Z level=info msg="Executing migration" id="create dashboard table" 2024-03-26T02:00:03.817 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.69643672Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-26T02:00:03.817 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.696950595Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-26T02:00:03.817 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.697473488Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-26T02:00:03.817 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.697948627Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-26T02:00:03.817 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.698466214Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-26T02:00:03.817 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.698939355Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-26T02:00:03.817 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.699899323Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-26T02:00:03.817 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.700385536Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-26T02:00:03.817 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.700901852Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-26T02:00:03.817 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.701427684Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-26T02:00:03.817 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.701839227Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-26T02:00:03.818 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.702336488Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-26T02:00:03.818 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.702697011Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-26T02:00:03.818 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.70327834Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-26T02:00:03.818 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.703864357Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-26T02:00:03.818 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.704444352Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-26T02:00:03.818 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.704936688Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-26T02:00:03.818 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.705509211Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-26T02:00:03.818 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.706073961Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-26T02:00:03.818 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.706575352Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-26T02:00:03.818 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.70693821Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-26T02:00:03.818 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.711232725Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-26T02:00:03.818 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.716121511Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-26T02:00:03.818 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.720860031Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-26T02:00:03.819 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.721474576Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-26T02:00:03.819 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.726230102Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-26T02:00:03.819 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.731381691Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-26T02:00:03.819 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.736450837Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-26T02:00:03.819 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.736919326Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-26T02:00:03.819 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.737257579Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-26T02:00:03.819 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.737709698Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-26T02:00:03.819 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.738204219Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-26T02:00:03.819 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.739163231Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-26T02:00:03.819 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.739608257Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-26T02:00:03.819 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.740099204Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-26T02:00:03.819 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.740641748Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-26T02:00:03.819 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.741018131Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-26T02:00:03.820 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.741415203Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-26T02:00:03.820 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.741926551Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-26T02:00:03.820 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.742400025Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-26T02:00:03.820 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.74272095Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-26T02:00:03.820 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.74307531Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-26T02:00:03.820 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.743530864Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-26T02:00:03.820 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.744114298Z level=info msg="Executing migration" id="create data_source table" 2024-03-26T02:00:03.820 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.744590824Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-26T02:00:03.820 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.745064674Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-26T02:00:03.820 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.745519597Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-26T02:00:03.820 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.745934381Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-26T02:00:03.820 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.746378129Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-26T02:00:03.820 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.747435919Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-26T02:00:03.820 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.747883061Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-26T02:00:03.821 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.748346807Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-26T02:00:03.821 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.748791105Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-26T02:00:03.821 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.74920378Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-26T02:00:03.821 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.749738309Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-26T02:00:03.821 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.750295879Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-26T02:00:03.821 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.750620858Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-26T02:00:03.821 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.750953401Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-26T02:00:03.821 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.751514454Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-26T02:00:03.821 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.751878905Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-26T02:00:03.821 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.752227892Z level=info msg="Executing migration" id="Add uid column" 2024-03-26T02:00:03.821 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.75276535Z level=info msg="Executing migration" id="Update uid value" 2024-03-26T02:00:03.821 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.753126626Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-26T02:00:03.821 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.753588752Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-26T02:00:03.821 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.754081413Z level=info msg="Executing migration" id="create api_key table" 2024-03-26T02:00:03.822 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.754601474Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-26T02:00:03.822 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.755126549Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-26T02:00:03.822 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.755631519Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-26T02:00:03.822 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.756176574Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-26T02:00:03.822 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.756660265Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-26T02:00:03.822 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.757163103Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-26T02:00:03.822 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.757651251Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-26T02:00:03.822 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.7587881Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-26T02:00:03.822 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.759282513Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-26T02:00:03.822 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.75975433Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-26T02:00:03.822 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.760260644Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-26T02:00:03.822 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.760768502Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-26T02:00:03.822 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.761190335Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-26T02:00:03.823 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.761610308Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-26T02:00:03.823 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.761962879Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-26T02:00:03.823 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.76260284Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-26T02:00:03.823 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.763233607Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-26T02:00:03.823 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.763631398Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-26T02:00:03.823 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.764288306Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-26T02:00:03.823 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.764918388Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-26T02:00:03.824 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.765398575Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-26T02:00:03.824 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.765869394Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-26T02:00:03.824 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.766354389Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-26T02:00:03.824 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.766851227Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-26T02:00:03.824 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.767361137Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-26T02:00:03.824 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.767859249Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-26T02:00:03.825 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.768247491Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-26T02:00:03.825 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.76861005Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-26T02:00:03.825 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.769262765Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-26T02:00:03.825 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.76990803Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-26T02:00:03.825 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.770311384Z level=info msg="Executing migration" id="create quota table v1" 2024-03-26T02:00:03.825 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.77078417Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-26T02:00:03.825 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.771312337Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-26T02:00:03.825 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.771664807Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-26T02:00:03.825 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.772223626Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-26T02:00:03.825 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.772737137Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-26T02:00:03.825 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.773393857Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-26T02:00:03.825 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.773774837Z level=info msg="Executing migration" id="create session table" 2024-03-26T02:00:03.825 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.77430539Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-26T02:00:03.826 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.774670271Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-26T02:00:03.826 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.775070463Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-26T02:00:03.826 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.775548599Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-26T02:00:03.826 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.776008037Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-26T02:00:03.826 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.77638611Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-26T02:00:03.826 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.776742582Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-26T02:00:03.826 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.777133211Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-26T02:00:03.826 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.777525268Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-26T02:00:03.826 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.777990056Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-26T02:00:03.826 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.778368525Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-26T02:00:03.826 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.779057483Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-26T02:00:03.826 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.779459412Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-26T02:00:03.826 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.78017063Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-26T02:00:03.827 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.780844502Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-26T02:00:03.827 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.781234759Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-26T02:00:03.827 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.781743376Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-26T02:00:03.827 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.782287404Z level=info msg="Executing migration" id="create alert table v1" 2024-03-26T02:00:03.827 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.78280922Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-26T02:00:03.827 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.7833566Z level=info msg="Executing migration" id="add index alert state" 2024-03-26T02:00:03.827 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.783858116Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-26T02:00:03.827 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.784358907Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-26T02:00:03.827 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.784810502Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-26T02:00:03.827 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.785355748Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-26T02:00:03.827 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.785783007Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-26T02:00:03.827 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.787223469Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-26T02:00:03.827 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.787642134Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-26T02:00:03.828 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.788103938Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-26T02:00:03.828 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.788450081Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-26T02:00:03.828 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.788809088Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-26T02:00:03.828 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.789228833Z level=info msg="Executing migration" id="Add column is_default" 2024-03-26T02:00:03.828 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.789868428Z level=info msg="Executing migration" id="Add column frequency" 2024-03-26T02:00:03.828 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.79054308Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-26T02:00:03.828 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.791213356Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-26T02:00:03.828 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.791874777Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-26T02:00:03.828 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.792322649Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-26T02:00:03.828 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.792612438Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-26T02:00:03.828 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.792905147Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-26T02:00:03.828 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.793325347Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-26T02:00:03.828 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.793778386Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-26T02:00:03.828 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.794226813Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-26T02:00:03.829 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.794624424Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-26T02:00:03.829 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.795205251Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-26T02:00:03.829 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.795882792Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-26T02:00:03.829 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.796556864Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-26T02:00:03.829 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.796886178Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-26T02:00:03.829 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.7973742Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-26T02:00:03.829 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.797791312Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-26T02:00:03.829 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.79848747Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-26T02:00:03.829 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.798797477Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-26T02:00:03.829 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.799289868Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-26T02:00:03.829 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.799766471Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-26T02:00:03.829 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.800167773Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-26T02:00:03.829 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.800638651Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-26T02:00:03.830 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.801111986Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-26T02:00:03.830 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.801616156Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-26T02:00:03.830 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.802168822Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-26T02:00:03.830 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.802716043Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-26T02:00:03.830 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.803284285Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-26T02:00:03.830 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.803669599Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-26T02:00:03.830 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.8044348Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-26T02:00:03.830 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.804946405Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-26T02:00:03.830 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.805729867Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-26T02:00:03.830 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.806226379Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-26T02:00:03.830 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.806758444Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-26T02:00:03.831 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.807297899Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-26T02:00:03.831 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.808975408Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-26T02:00:03.831 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.809461094Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-26T02:00:03.831 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.809999128Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-26T02:00:03.831 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.810424728Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-26T02:00:03.831 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.810931583Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-26T02:00:03.831 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.811391144Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-26T02:00:03.831 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.813818856Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-26T02:00:04.067 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.816249539Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-26T02:00:04.067 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.816754599Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-26T02:00:04.067 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.817316563Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-26T02:00:04.068 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.819763809Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-26T02:00:04.068 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.822241492Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-26T02:00:04.068 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.822694804Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-26T02:00:04.068 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.823093071Z level=info msg="Executing migration" id="Move region to single row" 2024-03-26T02:00:04.068 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.825593827Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-26T02:00:04.068 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.828024033Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-26T02:00:04.068 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.828491375Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-26T02:00:04.068 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.828969602Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-26T02:00:04.068 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.829607613Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-26T02:00:04.068 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.83144295Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-26T02:00:04.068 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.832117456Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-26T02:00:04.068 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.83263486Z level=info msg="Executing migration" id="create test_data table" 2024-03-26T02:00:04.069 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.833276203Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-26T02:00:04.069 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.833903851Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-26T02:00:04.069 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.83447641Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-26T02:00:04.069 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.83501344Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-26T02:00:04.069 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.835457443Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-26T02:00:04.069 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.835864289Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-26T02:00:04.069 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.836257878Z level=info msg="Executing migration" id="create team table" 2024-03-26T02:00:04.069 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.8367339Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-26T02:00:04.069 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.837313342Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-26T02:00:04.069 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.837880709Z level=info msg="Executing migration" id="create team member table" 2024-03-26T02:00:04.069 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.838394923Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-26T02:00:04.069 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.838936346Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-26T02:00:04.069 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.839486111Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-26T02:00:04.070 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.840027011Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-26T02:00:04.070 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.840848092Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-26T02:00:04.070 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.841662976Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-26T02:00:04.070 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.842492495Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-26T02:00:04.070 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.843011191Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-26T02:00:04.070 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.843584466Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-26T02:00:04.070 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.84416281Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-26T02:00:04.070 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.844715795Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-26T02:00:04.070 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.84528421Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-26T02:00:04.072 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.845808732Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-26T02:00:04.072 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.846363495Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-26T02:00:04.073 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.84692008Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-26T02:00:04.073 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.847428415Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-26T02:00:04.073 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.847870442Z level=info msg="Executing migration" id="create tag table" 2024-03-26T02:00:04.073 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.848372591Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-26T02:00:04.073 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.848917127Z level=info msg="Executing migration" id="create login attempt table" 2024-03-26T02:00:04.073 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.849406743Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-26T02:00:04.073 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.849934963Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-26T02:00:04.073 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.850453244Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-26T02:00:04.073 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.852432991Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-26T02:00:04.073 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.852923241Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-26T02:00:04.073 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.85345383Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-26T02:00:04.073 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.85386946Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-26T02:00:04.073 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.854342991Z level=info msg="Executing migration" id="create user auth table" 2024-03-26T02:00:04.073 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.854815445Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-26T02:00:04.074 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.855353535Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-26T02:00:04.074 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.855722511Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-26T02:00:04.074 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.856587294Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-26T02:00:04.074 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.857447459Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-26T02:00:04.074 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.858293427Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-26T02:00:04.074 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.859156895Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-26T02:00:04.074 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.859680734Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-26T02:00:04.074 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.860547855Z level=info msg="Executing migration" id="create server_lock table" 2024-03-26T02:00:04.074 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.8610489Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-26T02:00:04.074 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.861581637Z level=info msg="Executing migration" id="create user auth token table" 2024-03-26T02:00:04.074 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.862126497Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-26T02:00:04.074 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.86264505Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-26T02:00:04.075 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.863176007Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-26T02:00:04.075 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.863694752Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-26T02:00:04.075 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.864588039Z level=info msg="Executing migration" id="create cache_data table" 2024-03-26T02:00:04.075 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.865104381Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-26T02:00:04.075 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.865627842Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-26T02:00:04.075 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.866163906Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-26T02:00:04.075 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.866681071Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-26T02:00:04.075 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.867084637Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-26T02:00:04.075 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.867501202Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-26T02:00:04.075 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.868069357Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-26T02:00:04.075 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.868632479Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-26T02:00:04.075 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.86922795Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-26T02:00:04.075 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.869570714Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-26T02:00:04.076 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.870112088Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-26T02:00:04.076 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.870638474Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-26T02:00:04.076 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.871183989Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-26T02:00:04.076 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.871720788Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-26T02:00:04.076 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.872577227Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-26T02:00:04.076 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.873105822Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-26T02:00:04.076 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.873462251Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-26T02:00:04.076 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.873894603Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-26T02:00:04.076 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.874424319Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-26T02:00:04.076 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.874917323Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-26T02:00:04.076 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.875255737Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-26T02:00:04.076 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.87569929Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-26T02:00:04.076 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.876162839Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-26T02:00:04.076 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.876640622Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-26T02:00:04.077 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.877122297Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-26T02:00:04.077 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.877951205Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-26T02:00:04.077 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.878447822Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-26T02:00:04.077 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.87889258Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-26T02:00:04.077 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.882381262Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-26T02:00:04.077 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.885882221Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-26T02:00:04.077 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.88636896Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-26T02:00:04.077 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.88680439Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-26T02:00:04.077 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.887715757Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-26T02:00:04.077 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.88821206Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-26T02:00:04.077 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.88890394Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-26T02:00:04.077 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.889400014Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-26T02:00:04.077 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.889896413Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-26T02:00:04.078 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.890237078Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-26T02:00:04.078 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.891099481Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-26T02:00:04.078 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.891943065Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-26T02:00:04.078 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.892784889Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-26T02:00:04.078 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.893262683Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-26T02:00:04.078 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.89371697Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-26T02:00:04.078 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.894557297Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-26T02:00:04.078 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.895415761Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-26T02:00:04.078 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.895923437Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-26T02:00:04.078 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.896807894Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-26T02:00:04.078 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.897669411Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-26T02:00:04.078 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.898047334Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-26T02:00:04.078 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.898540638Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-26T02:00:04.079 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.899060449Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-26T02:00:04.079 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.899574584Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-26T02:00:04.079 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.899886483Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-26T02:00:04.079 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.900821279Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-26T02:00:04.079 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.901701262Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-26T02:00:04.079 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.902571216Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-26T02:00:04.079 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.903450417Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-26T02:00:04.079 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.904341041Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-26T02:00:04.079 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.90469384Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-26T02:00:04.079 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.905170701Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-26T02:00:04.079 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.906044853Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-26T02:00:04.079 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.906431147Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-26T02:00:04.079 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.90735474Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-26T02:00:04.079 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.90786203Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-26T02:00:04.080 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.908775283Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-26T02:00:04.080 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.909238962Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-26T02:00:04.080 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.909723811Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-26T02:00:04.080 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.910628564Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-26T02:00:04.080 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.9110928Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-26T02:00:04.080 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.911646638Z level=info msg="Executing migration" id="create alert_image table" 2024-03-26T02:00:04.080 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.912139902Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-26T02:00:04.080 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.91588465Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-26T02:00:04.080 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.920348018Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-26T02:00:04.080 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.924954694Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-26T02:00:04.080 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.925573189Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T02:00:04.080 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.925659169Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T02:00:04.080 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.925950434Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-26T02:00:04.080 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.927413188Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-26T02:00:04.081 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.929354767Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-26T02:00:04.081 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.929481019Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-26T02:00:04.081 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.929593638Z level=info msg="alerts found to migrate" alerts=0 2024-03-26T02:00:04.081 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.93130346Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-26T02:00:04.081 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.933234044Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-26T02:00:04.081 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.935235899Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-26T02:00:04.081 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.937245376Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-26T02:00:04.081 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.939204117Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-26T02:00:04.081 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.939912013Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-26T02:00:04.081 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.941843011Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-26T02:00:04.081 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.942322887Z level=info msg="Executing migration" id="create data_keys table" 2024-03-26T02:00:04.081 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.942852431Z level=info msg="Executing migration" id="create secrets table" 2024-03-26T02:00:04.081 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.943383212Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-26T02:00:04.082 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.947860229Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-26T02:00:04.082 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.94887055Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-26T02:00:04.082 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.949291712Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-26T02:00:04.082 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.953698514Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-26T02:00:04.082 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.958142473Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-26T02:00:04.082 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.958647307Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-26T02:00:04.082 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.959192948Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-26T02:00:04.082 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.959624674Z level=info msg="Executing migration" id="create permission table" 2024-03-26T02:00:04.082 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.960155059Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-26T02:00:04.082 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.960682035Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-26T02:00:04.082 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.961227949Z level=info msg="Executing migration" id="create role table" 2024-03-26T02:00:04.082 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.961711005Z level=info msg="Executing migration" id="add column display_name" 2024-03-26T02:00:04.082 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.962722787Z level=info msg="Executing migration" id="add column group_name" 2024-03-26T02:00:04.082 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.963726079Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-26T02:00:04.083 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.964276444Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-26T02:00:04.083 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.964800141Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-26T02:00:04.083 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.965350567Z level=info msg="Executing migration" id="create team role table" 2024-03-26T02:00:04.083 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.965821301Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-26T02:00:04.083 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.966403557Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-26T02:00:04.083 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.966948661Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-26T02:00:04.083 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.967485102Z level=info msg="Executing migration" id="create user role table" 2024-03-26T02:00:04.083 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.967961243Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-26T02:00:04.083 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.968513169Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-26T02:00:04.083 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.969047181Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-26T02:00:04.083 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.969574301Z level=info msg="Executing migration" id="create builtin role table" 2024-03-26T02:00:04.083 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.970109318Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-26T02:00:04.083 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.970644342Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-26T02:00:04.083 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.971178643Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-26T02:00:04.084 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.972246804Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-26T02:00:04.084 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.972822803Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-26T02:00:04.084 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.97336462Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-26T02:00:04.084 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.973861949Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-26T02:00:04.084 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.974386555Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-26T02:00:04.084 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.974862462Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-26T02:00:04.084 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.975402407Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-26T02:00:04.084 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.976463767Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-26T02:00:04.084 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.976958657Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-26T02:00:04.084 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.977528636Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-26T02:00:04.084 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.977891577Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-26T02:00:04.084 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.978428184Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-26T02:00:04.084 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.978951204Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-26T02:00:04.085 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.979433907Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-26T02:00:04.085 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.979781133Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-26T02:00:04.085 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.980195706Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-26T02:00:04.085 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.980606838Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-26T02:00:04.085 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.98109005Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-26T02:00:04.085 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.982098212Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-26T02:00:04.085 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.982426054Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-26T02:00:04.085 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.982863426Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-26T02:00:04.085 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.983331919Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-26T02:00:04.085 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.983792655Z level=info msg="Executing migration" id="add correlation config column" 2024-03-26T02:00:04.085 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.984820422Z level=info msg="Executing migration" id="create entity_events table" 2024-03-26T02:00:04.085 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.985331627Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-26T02:00:04.085 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.985761913Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T02:00:04.085 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.985860486Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T02:00:04.086 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.986173997Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T02:00:04.086 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.986269528Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T02:00:04.086 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.986636487Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-26T02:00:04.086 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.987160142Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-26T02:00:04.086 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.987665555Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-26T02:00:04.086 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.988247726Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T02:00:04.086 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.988794665Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-26T02:00:04.086 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.989338218Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T02:00:04.086 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.989856036Z level=info msg="Executing migration" id="Drop public config table" 2024-03-26T02:00:04.086 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.990376914Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-26T02:00:04.086 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.990869865Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-26T02:00:04.086 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.991414168Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T02:00:04.086 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.991948993Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-26T02:00:04.087 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.992494655Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-26T02:00:04.087 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.995612729Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-26T02:00:04.087 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.99673318Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-26T02:00:04.087 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.997846465Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-26T02:00:04.087 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.998287312Z level=info msg="Executing migration" id="add share column" 2024-03-26T02:00:04.087 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.999396431Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-26T02:00:04.087 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:03 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:03.999944851Z level=info msg="Executing migration" id="create file table" 2024-03-26T02:00:04.087 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.000483531Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-26T02:00:04.087 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.001025574Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-26T02:00:04.087 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.001547251Z level=info msg="Executing migration" id="create file_meta table" 2024-03-26T02:00:04.087 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.002031776Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-26T02:00:04.087 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.002578641Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-26T02:00:04.087 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.002939413Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-26T02:00:04.087 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.003413146Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-26T02:00:04.088 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.003809364Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-26T02:00:04.088 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.004361273Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-26T02:00:04.088 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.005473584Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-26T02:00:04.088 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.005904506Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-26T02:00:04.088 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.006463838Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-26T02:00:04.088 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.006929826Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-26T02:00:04.088 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.007352589Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-26T02:00:04.088 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.007796763Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-26T02:00:04.088 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.008915974Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-26T02:00:04.088 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.010027221Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-26T02:00:04.088 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.010571897Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-26T02:00:04.088 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.020736722Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-26T02:00:04.088 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.021312698Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-26T02:00:04.088 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.021869966Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-26T02:00:04.089 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.025503561Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-26T02:00:04.089 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.025927883Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-26T02:00:04.089 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.026388569Z level=info msg="Executing migration" id="create folder table" 2024-03-26T02:00:04.089 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.026863507Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-26T02:00:04.089 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.027412136Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-26T02:00:04.089 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.027978289Z level=info msg="Executing migration" id="Update folder title length" 2024-03-26T02:00:04.089 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.028396898Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-26T02:00:04.089 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=migrator t=2024-03-26T02:00:04.028903652Z level=info msg="migrations completed" performed=485 skipped=0 duration=367.928902ms 2024-03-26T02:00:04.089 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=sqlstore t=2024-03-26T02:00:04.029299132Z level=info msg="Created default organization" 2024-03-26T02:00:04.520 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=plugin.loader t=2024-03-26T02:00:04.101024879Z level=info msg="Plugin registered" pluginID=input 2024-03-26T02:00:04.520 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=plugin.loader t=2024-03-26T02:00:04.131435294Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-26T02:00:04.520 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=plugin.loader t=2024-03-26T02:00:04.131471075Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-26T02:00:04.520 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=secrets t=2024-03-26T02:00:04.131507194Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-26T02:00:04.520 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=query_data t=2024-03-26T02:00:04.133682294Z level=info msg="Query Service initialization" 2024-03-26T02:00:04.520 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=live.push_http t=2024-03-26T02:00:04.136019887Z level=info msg="Live Push Gateway initialization" 2024-03-26T02:00:05.270 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=infra.usagestats.collector t=2024-03-26T02:00:04.781228508Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-26T02:00:05.270 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=provisioning.datasources t=2024-03-26T02:00:04.782347473Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-26T02:00:05.270 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=provisioning.datasources t=2024-03-26T02:00:04.793738615Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-26T02:00:05.270 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=provisioning.alerting t=2024-03-26T02:00:04.805019732Z level=info msg="starting to provision alerting" 2024-03-26T02:00:05.270 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=provisioning.alerting t=2024-03-26T02:00:04.805039083Z level=info msg="finished to provision alerting" 2024-03-26T02:00:05.270 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=ngalert.state.manager t=2024-03-26T02:00:04.805127971Z level=info msg="Warming state cache for startup" 2024-03-26T02:00:05.270 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=ngalert.state.manager t=2024-03-26T02:00:04.805323051Z level=info msg="State cache has been initialized" states=0 duration=194.252µs 2024-03-26T02:00:05.270 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=ticker t=2024-03-26T02:00:04.805410885Z level=info msg=starting first_tick=2024-03-26T02:00:10Z 2024-03-26T02:00:05.270 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=grafanaStorageLogger t=2024-03-26T02:00:04.805142074Z level=info msg="storage starting" 2024-03-26T02:00:05.270 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=ngalert.multiorg.alertmanager t=2024-03-26T02:00:04.805450577Z level=info msg="starting MultiOrg Alertmanager" 2024-03-26T02:00:05.271 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=http.server t=2024-03-26T02:00:04.806865223Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-26T02:00:05.271 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=sqlstore.transactions t=2024-03-26T02:00:04.820616347Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T02:00:05.271 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=sqlstore.transactions t=2024-03-26T02:00:04.82142507Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T02:00:05.271 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:04 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a[44027]: logger=sqlstore.transactions t=2024-03-26T02:00:04.830925551Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-26T02:00:05.589 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 02:00:05 smithi055 podman[43763]: 2024-03-26 02:00:05.310189919 +0000 UTC m=+5.531297097 container died 46926b4403bf132c8b7c3e302a670b32c7668346b2f10a0f6f9166a0461de989 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-26T02:00:05.855 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 02:00:05 smithi055 podman[43763]: 2024-03-26 02:00:05.679811891 +0000 UTC m=+5.900919062 container cleanup 46926b4403bf132c8b7c3e302a670b32c7668346b2f10a0f6f9166a0461de989 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-26T02:00:05.857 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 02:00:05 smithi055 bash[43763]: ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-1 2024-03-26T02:00:06.269 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 02:00:05 smithi055 podman[44078]: 2024-03-26 02:00:05.855971357 +0000 UTC m=+0.541793219 container remove 46926b4403bf132c8b7c3e302a670b32c7668346b2f10a0f6f9166a0461de989 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-1, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-26T02:00:06.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 02:00:06 smithi055 podman[44174]: 2024-03-26 02:00:06.199447801 +0000 UTC m=+0.017811713 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:00:06.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 02:00:06 smithi055 podman[44174]: 2024-03-26 02:00:06.314977351 +0000 UTC m=+0.133341237 container create 6d0b6afc518497cf28b37ee83029ffce7fd4c200ecc4c10236ff90459f8e8821 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-26T02:00:06.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 02:00:06 smithi055 podman[44174]: 2024-03-26 02:00:06.526011709 +0000 UTC m=+0.344375607 container init 6d0b6afc518497cf28b37ee83029ffce7fd4c200ecc4c10236ff90459f8e8821 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-26T02:00:06.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 02:00:06 smithi055 podman[44174]: 2024-03-26 02:00:06.531192683 +0000 UTC m=+0.349556568 container start 6d0b6afc518497cf28b37ee83029ffce7fd4c200ecc4c10236ff90459f8e8821 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:00:06.626 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 02:00:06 smithi055 podman[44174]: 2024-03-26 02:00:06.625743909 +0000 UTC m=+0.444107794 container attach 6d0b6afc518497cf28b37ee83029ffce7fd4c200ecc4c10236ff90459f8e8821 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-03-26T02:00:07.020 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 02:00:06 smithi055 podman[44174]: 2024-03-26 02:00:06.832653493 +0000 UTC m=+0.651017383 container died 6d0b6afc518497cf28b37ee83029ffce7fd4c200ecc4c10236ff90459f8e8821 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240319, ceph=True, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-26T02:00:07.357 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@osd.1.service' 2024-03-26T02:00:07.399 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:00:07.399 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-26T02:00:07.399 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-26T02:00:07.399 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-b95eae18-eb13-11ee-95d0-87774f69a715@osd.2 2024-03-26T02:00:07.715 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:00:07 smithi066 systemd[1]: Stopping Ceph osd.2 for b95eae18-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:00:07.977 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:00:07 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-2[35524]: 2024-03-26T02:00:07.714+0000 7f80735d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:00:07.977 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:00:07 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-2[35524]: 2024-03-26T02:00:07.714+0000 7f80735d3700 -1 osd.2 32 *** Got signal Terminated *** 2024-03-26T02:00:07.978 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:00:07 smithi066 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-2[35524]: 2024-03-26T02:00:07.714+0000 7f80735d3700 -1 osd.2 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:00:13.477 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:00:13 smithi066 podman[44293]: 2024-03-26 02:00:13.223937084 +0000 UTC m=+5.621706311 container died 9a6d5e087cb2590be4edbac4aed01c1f401c27277fa023a5558622327cf50b36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-2, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-26T02:00:13.936 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:00:13 smithi066 podman[44293]: 2024-03-26 02:00:13.702960176 +0000 UTC m=+6.100729393 container cleanup 9a6d5e087cb2590be4edbac4aed01c1f401c27277fa023a5558622327cf50b36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-26T02:00:13.937 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:00:13 smithi066 bash[44293]: ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-2 2024-03-26T02:00:14.227 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:00:13 smithi066 podman[44315]: 2024-03-26 02:00:13.937019574 +0000 UTC m=+0.715024070 container remove 9a6d5e087cb2590be4edbac4aed01c1f401c27277fa023a5558622327cf50b36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:00:14.548 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:00:14 smithi066 podman[44411]: 2024-03-26 02:00:14.271296865 +0000 UTC m=+0.108704455 container create dcab59b7769a3334bc02c2db4bf819390db5b3ffa180b25d394b4f560e4d21c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-03-26T02:00:14.548 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:00:14 smithi066 podman[44411]: 2024-03-26 02:00:14.180654067 +0000 UTC m=+0.018061628 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:00:14.548 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:00:14 smithi066 podman[44411]: 2024-03-26 02:00:14.465501491 +0000 UTC m=+0.302909047 container init dcab59b7769a3334bc02c2db4bf819390db5b3ffa180b25d394b4f560e4d21c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-2-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-5e70124, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-26T02:00:14.548 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:00:14 smithi066 podman[44411]: 2024-03-26 02:00:14.470948094 +0000 UTC m=+0.308355642 container start dcab59b7769a3334bc02c2db4bf819390db5b3ffa180b25d394b4f560e4d21c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:00:14.548 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:00:14 smithi066 podman[44411]: 2024-03-26 02:00:14.548264693 +0000 UTC m=+0.385672248 container attach dcab59b7769a3334bc02c2db4bf819390db5b3ffa180b25d394b4f560e4d21c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-26T02:00:14.977 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:00:14 smithi066 podman[44411]: 2024-03-26 02:00:14.777977934 +0000 UTC m=+0.615385482 container died dcab59b7769a3334bc02c2db4bf819390db5b3ffa180b25d394b4f560e4d21c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:00:15.386 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@osd.2.service' 2024-03-26T02:00:15.429 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:00:15.429 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-26T02:00:15.430 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-26T02:00:15.430 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-b95eae18-eb13-11ee-95d0-87774f69a715@rgw.foo.a 2024-03-26T02:00:15.760 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 26 02:00:15 smithi003 systemd[1]: Stopping Ceph rgw.foo.a for b95eae18-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:00:16.032 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 26 02:00:15 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-rgw-foo-a[46898]: 2024-03-26T02:00:15.759+0000 7fc6ee5bf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:00:16.032 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 26 02:00:15 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-rgw-foo-a[46898]: 2024-03-26T02:00:15.759+0000 7fc6fce3cac0 -1 shutting down 2024-03-26T02:00:26.097 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 26 02:00:25 smithi003 bash[56649]: time="2024-03-26T02:00:25Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-b95eae18-eb13-11ee-95d0-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-26T02:00:26.098 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 26 02:00:25 smithi003 podman[56649]: 2024-03-26 02:00:25.78691044 +0000 UTC m=+10.150348262 container died 961bfb838631da66ab03242e8bc719af43b123a5f5b4bee22b1ef2bda67c8f7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-rgw-foo-a, RELEASE=squid-5e70124, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:00:26.641 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 26 02:00:26 smithi003 podman[56649]: 2024-03-26 02:00:26.387559566 +0000 UTC m=+10.750997387 container cleanup 961bfb838631da66ab03242e8bc719af43b123a5f5b4bee22b1ef2bda67c8f7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-rgw-foo-a, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:00:26.641 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 26 02:00:26 smithi003 bash[56649]: ceph-b95eae18-eb13-11ee-95d0-87774f69a715-rgw-foo-a 2024-03-26T02:00:26.641 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 26 02:00:26 smithi003 podman[56685]: 2024-03-26 02:00:26.605762005 +0000 UTC m=+0.811488744 container remove 961bfb838631da66ab03242e8bc719af43b123a5f5b4bee22b1ef2bda67c8f7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-rgw-foo-a, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:00:26.641 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 26 02:00:26 smithi003 systemd[1]: ceph-b95eae18-eb13-11ee-95d0-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-26T02:00:26.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@rgw.foo.a.service' 2024-03-26T02:00:26.758 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:00:26.759 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-26T02:00:26.759 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-26T02:00:26.759 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-b95eae18-eb13-11ee-95d0-87774f69a715@prometheus.a 2024-03-26T02:00:27.058 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 02:00:26 smithi055 systemd[1]: Stopping Ceph prometheus.a for b95eae18-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:00:27.348 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 02:00:27 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T02:00:27.058Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T02:00:27.348 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 02:00:27 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T02:00:27.058Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T02:00:27.348 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 02:00:27 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T02:00:27.058Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T02:00:27.348 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 02:00:27 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T02:00:27.058Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T02:00:27.348 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 02:00:27 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T02:00:27.058Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T02:00:27.348 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 02:00:27 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T02:00:27.058Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T02:00:27.348 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 02:00:27 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T02:00:27.058Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T02:00:27.348 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 02:00:27 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T02:00:27.058Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T02:00:27.348 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 02:00:27 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T02:00:27.058Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T02:00:27.349 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 02:00:27 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T02:00:27.058Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T02:00:27.349 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 02:00:27 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T02:00:27.059Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T02:00:27.349 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 02:00:27 smithi055 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a[41783]: ts=2024-03-26T02:00:27.059Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T02:00:27.349 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 02:00:27 smithi055 podman[44308]: 2024-03-26 02:00:27.068946769 +0000 UTC m=+0.116720649 container died d99a45fedd93fa24f72145ddf17866204bf1d09d401ceea119b00fe67d3de245 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:00:27.698 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 02:00:27 smithi055 podman[44308]: 2024-03-26 02:00:27.438904778 +0000 UTC m=+0.486678663 container cleanup d99a45fedd93fa24f72145ddf17866204bf1d09d401ceea119b00fe67d3de245 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:00:27.698 INFO:journalctl@ceph.prometheus.a.smithi055.stdout:Mar 26 02:00:27 smithi055 bash[44308]: ceph-b95eae18-eb13-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T02:00:27.869 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b95eae18-eb13-11ee-95d0-87774f69a715@prometheus.a.service' 2024-03-26T02:00:27.906 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:00:27.906 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-26T02:00:27.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b95eae18-eb13-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T02:00:28.102 INFO:teuthology.orchestra.run.smithi003.stdout:Deleting cluster with fsid: b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T02:00:29.141 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 02:00:28 smithi003 systemd[1]: Stopping Ceph alertmanager.a for b95eae18-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:00:29.442 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 02:00:29 smithi003 ceph-b95eae18-eb13-11ee-95d0-87774f69a715-alertmanager-a[54138]: ts=2024-03-26T02:00:29.178Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T02:00:29.443 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 02:00:29 smithi003 podman[57002]: 2024-03-26 02:00:29.199355898 +0000 UTC m=+0.124394965 container died 808d9b2a0f555b45bc96824f2452340443c8435ee772927fa8e9315100a44cf2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:00:29.784 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 02:00:29 smithi003 podman[57002]: 2024-03-26 02:00:29.617464667 +0000 UTC m=+0.542503772 container cleanup 808d9b2a0f555b45bc96824f2452340443c8435ee772927fa8e9315100a44cf2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:00:29.785 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 02:00:29 smithi003 bash[57002]: ceph-b95eae18-eb13-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T02:00:30.075 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 02:00:29 smithi003 podman[57022]: 2024-03-26 02:00:29.785049652 +0000 UTC m=+0.586011076 container remove 808d9b2a0f555b45bc96824f2452340443c8435ee772927fa8e9315100a44cf2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:00:30.075 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 02:00:29 smithi003 podman[57022]: 2024-03-26 02:00:29.852304978 +0000 UTC m=+0.653266402 volume remove 682a33a4acfa52aea6ee1496f2bdeebdbf5b2c1ee788b2f8a08371008daf08fc 2024-03-26T02:00:30.390 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 02:00:30 smithi003 systemd[1]: ceph-b95eae18-eb13-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T02:00:30.391 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 26 02:00:30 smithi003 systemd[1]: Stopped Ceph alertmanager.a for b95eae18-eb13-11ee-95d0-87774f69a715. 2024-03-26T02:00:30.720 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b95eae18-eb13-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T02:00:30.917 INFO:teuthology.orchestra.run.smithi055.stdout:Deleting cluster with fsid: b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T02:00:33.020 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:32 smithi055 systemd[1]: Stopping Ceph grafana.a for b95eae18-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:00:34.269 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:33 smithi055 podman[44659]: 2024-03-26 02:00:33.857785905 +0000 UTC m=+0.983102490 container died 61c884ae3291fefb1cae93917d7af8050da08b4fd7314484cfcd0407d025bb57 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, name=ubi8, description=Ceph Grafana Container, version=8.5, vcs-type=git, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, maintainer=Paul Cuzner , vendor=Red Hat, Inc., io.buildah.version=1.32.0) 2024-03-26T02:00:36.020 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:35 smithi055 podman[44659]: 2024-03-26 02:00:35.840502427 +0000 UTC m=+2.965819011 container cleanup 61c884ae3291fefb1cae93917d7af8050da08b4fd7314484cfcd0407d025bb57 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a, architecture=x86_64, name=ubi8, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.tags=base rhel8, build-date=2022-04-29T11:48:11.055897, vcs-type=git, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=239.1651231664, version=8.5, distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , vendor=Red Hat, Inc.) 2024-03-26T02:00:36.020 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:35 smithi055 bash[44659]: ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a 2024-03-26T02:00:36.361 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:36 smithi055 podman[44679]: 2024-03-26 02:00:36.100118456 +0000 UTC m=+2.240120584 container remove 61c884ae3291fefb1cae93917d7af8050da08b4fd7314484cfcd0407d025bb57 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-b95eae18-eb13-11ee-95d0-87774f69a715-grafana-a, com.redhat.component=ubi8-container, version=8.5, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., build-date=2022-04-29T11:48:11.055897, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-03-26T02:00:36.362 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:36 smithi055 systemd[1]: ceph-b95eae18-eb13-11ee-95d0-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T02:00:36.362 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:36 smithi055 systemd[1]: ceph-b95eae18-eb13-11ee-95d0-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-26T02:00:36.362 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:36 smithi055 systemd[1]: Stopped Ceph grafana.a for b95eae18-eb13-11ee-95d0-87774f69a715. 2024-03-26T02:00:36.362 INFO:journalctl@ceph.grafana.a.smithi055.stdout:Mar 26 02:00:36 smithi055 systemd[1]: ceph-b95eae18-eb13-11ee-95d0-87774f69a715@grafana.a.service: Consumed 2.599s CPU time. 2024-03-26T02:00:36.853 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b95eae18-eb13-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T02:00:37.056 INFO:teuthology.orchestra.run.smithi066.stdout:Deleting cluster with fsid: b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T02:00:38.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:00:38.261 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:00:38.296 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:00:38.329 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-26T02:00:38.331 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621927/remote/smithi003/crash 2024-03-26T02:00:38.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/crash -- . 2024-03-26T02:00:38.425 INFO:teuthology.orchestra.run.smithi003.stderr:tar: /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T02:00:38.426 INFO:teuthology.orchestra.run.smithi003.stderr:tar: Error is not recoverable: exiting now 2024-03-26T02:00:38.427 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621927/remote/smithi055/crash 2024-03-26T02:00:38.428 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/crash -- . 2024-03-26T02:00:38.473 INFO:teuthology.orchestra.run.smithi055.stderr:tar: /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T02:00:38.473 INFO:teuthology.orchestra.run.smithi055.stderr:tar: Error is not recoverable: exiting now 2024-03-26T02:00:38.475 DEBUG:teuthology.misc:Transferring archived files from smithi066:/var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621927/remote/smithi066/crash 2024-03-26T02:00:38.476 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/crash -- . 2024-03-26T02:00:38.561 INFO:teuthology.orchestra.run.smithi066.stderr:tar: /var/lib/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T02:00:38.561 INFO:teuthology.orchestra.run.smithi066.stderr:tar: Error is not recoverable: exiting now 2024-03-26T02:00:38.562 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-26T02:00:38.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T02:00:38.595 INFO:teuthology.orchestra.run.smithi003.stdout:2024-03-26T01:56:48.619706+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T02:00:38.595 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-26T02:00:38.595 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[SEC\]' /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T02:00:38.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[ERR\]' /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T02:00:38.736 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[WRN\]' /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T02:00:38.807 INFO:teuthology.orchestra.run.smithi003.stdout:2024-03-26T01:56:48.619706+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T02:00:38.807 INFO:tasks.cephadm:Compressing logs... 2024-03-26T02:00:38.807 DEBUG:teuthology.orchestra.run.smithi003:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T02:00:38.850 DEBUG:teuthology.orchestra.run.smithi055:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T02:00:38.853 DEBUG:teuthology.orchestra.run.smithi066:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T02:00:38.874 INFO:teuthology.orchestra.run.smithi003.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T02:00:38.875 INFO:teuthology.orchestra.run.smithi003.stderr:: No such file or directory 2024-03-26T02:00:38.876 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-26T02:00:38.876 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T02:00:38.876 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-26T02:00:38.876 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-mon.a.log: /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-26T02:00:38.876 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-26T02:00:38.876 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log: /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-mgr.y.log: 57.9% -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-26T02:00:38.877 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T02:00:38.878 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.log 2024-03-26T02:00:38.878 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.cephadm.log: 90.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T02:00:38.878 INFO:teuthology.orchestra.run.smithi003.stderr: 89.3% -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T02:00:38.878 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T02:00:38.878 INFO:teuthology.orchestra.run.smithi003.stderr: 82.6%/var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.log: -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T02:00:38.878 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-volume.log: 84.6% -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T02:00:38.881 INFO:teuthology.orchestra.run.smithi055.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T02:00:38.881 INFO:teuthology.orchestra.run.smithi055.stderr:: No such file or directory 2024-03-26T02:00:38.881 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T02:00:38.882 INFO:teuthology.orchestra.run.smithi003.stderr: 90.5% -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T02:00:38.883 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-osd.1.log/var/log/ceph/cephadm.log: 2024-03-26T02:00:38.883 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-26T02:00:38.883 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.audit.log: /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T02:00:38.883 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.log 2024-03-26T02:00:38.884 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-mgr.x.log: /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.cephadm.log: 89.4% -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T02:00:38.884 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T02:00:38.885 INFO:teuthology.orchestra.run.smithi055.stderr: 81.2% -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T02:00:38.885 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-mon.b.log/var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.log: 2024-03-26T02:00:38.885 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-volume.log: 83.9% -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T02:00:38.885 INFO:teuthology.orchestra.run.smithi055.stderr: 81.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T02:00:38.886 INFO:teuthology.orchestra.run.smithi066.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T02:00:38.886 INFO:teuthology.orchestra.run.smithi066.stderr:: No such file or directory 2024-03-26T02:00:38.886 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T02:00:38.886 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-mgr.z.log 2024-03-26T02:00:38.886 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T02:00:38.888 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.audit.log: /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.log 2024-03-26T02:00:38.888 INFO:teuthology.orchestra.run.smithi066.stderr: 90.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T02:00:38.888 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-26T02:00:38.888 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.cephadm.log: 81.2% -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T02:00:38.888 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.log: 89.4% -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T02:00:38.888 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T02:00:38.888 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-26T02:00:38.889 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-osd.2.log: /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-volume.log: 84.1% -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T02:00:38.889 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-mon.b.log: 90.1% -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-26T02:00:38.889 INFO:teuthology.orchestra.run.smithi055.stderr: 90.5% -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T02:00:38.890 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-mon.c.log: 90.1% -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-mgr.z.log.gz 2024-03-26T02:00:38.890 INFO:teuthology.orchestra.run.smithi066.stderr: 90.5% -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T02:00:38.939 INFO:teuthology.orchestra.run.smithi055.stderr: 92.2% -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-26T02:00:38.942 INFO:teuthology.orchestra.run.smithi066.stderr: 92.2% -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-26T02:00:38.943 INFO:teuthology.orchestra.run.smithi003.stderr: 89.0% -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-26T02:00:39.048 INFO:teuthology.orchestra.run.smithi003.stderr: 91.3% -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-26T02:00:39.169 INFO:teuthology.orchestra.run.smithi003.stderr: 94.2% -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-26T02:00:39.171 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-26T02:00:39.171 INFO:teuthology.orchestra.run.smithi003.stderr:real 0m0.309s 2024-03-26T02:00:39.171 INFO:teuthology.orchestra.run.smithi003.stderr:user 0m0.536s 2024-03-26T02:00:39.171 INFO:teuthology.orchestra.run.smithi003.stderr:sys 0m0.036s 2024-03-26T02:00:39.190 INFO:teuthology.orchestra.run.smithi066.stderr: 94.2% -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-26T02:00:39.192 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-26T02:00:39.192 INFO:teuthology.orchestra.run.smithi066.stderr:real 0m0.324s 2024-03-26T02:00:39.192 INFO:teuthology.orchestra.run.smithi066.stderr:user 0m0.377s 2024-03-26T02:00:39.192 INFO:teuthology.orchestra.run.smithi066.stderr:sys 0m0.033s 2024-03-26T02:00:39.221 INFO:teuthology.orchestra.run.smithi055.stderr: 94.2% -- replaced with /var/log/ceph/b95eae18-eb13-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-26T02:00:39.223 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-26T02:00:39.223 INFO:teuthology.orchestra.run.smithi055.stderr:real 0m0.356s 2024-03-26T02:00:39.223 INFO:teuthology.orchestra.run.smithi055.stderr:user 0m0.403s 2024-03-26T02:00:39.224 INFO:teuthology.orchestra.run.smithi055.stderr:sys 0m0.041s 2024-03-26T02:00:39.224 INFO:tasks.cephadm:Archiving logs... 2024-03-26T02:00:39.224 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621927/remote/smithi003/log 2024-03-26T02:00:39.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T02:00:39.344 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621927/remote/smithi055/log 2024-03-26T02:00:39.345 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T02:00:39.433 DEBUG:teuthology.misc:Transferring archived files from smithi066:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621927/remote/smithi066/log 2024-03-26T02:00:39.434 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T02:00:39.521 INFO:tasks.cephadm:Removing cluster... 2024-03-26T02:00:39.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b95eae18-eb13-11ee-95d0-87774f69a715 --force 2024-03-26T02:00:39.706 INFO:teuthology.orchestra.run.smithi003.stdout:Deleting cluster with fsid: b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T02:00:39.819 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b95eae18-eb13-11ee-95d0-87774f69a715 --force 2024-03-26T02:00:40.010 INFO:teuthology.orchestra.run.smithi055.stdout:Deleting cluster with fsid: b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T02:00:40.124 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b95eae18-eb13-11ee-95d0-87774f69a715 --force 2024-03-26T02:00:40.326 INFO:teuthology.orchestra.run.smithi066.stdout:Deleting cluster with fsid: b95eae18-eb13-11ee-95d0-87774f69a715 2024-03-26T02:00:40.446 INFO:tasks.cephadm:Removing cephadm ... 2024-03-26T02:00:40.447 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T02:00:40.464 DEBUG:teuthology.orchestra.run.smithi055:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T02:00:40.479 DEBUG:teuthology.orchestra.run.smithi066:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T02:00:40.494 INFO:tasks.cephadm:Teardown complete 2024-03-26T02:00:40.495 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-26T02:00:40.504 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi003:/dev/vg_nvme/lv_1... 2024-03-26T02:00:40.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo nvme disconnect -n lv_1 2024-03-26T02:00:40.652 INFO:teuthology.orchestra.run.smithi003.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-26T02:00:40.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:00:40.653 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi003:/dev/vg_nvme/lv_2... 2024-03-26T02:00:40.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo nvme disconnect -n lv_2 2024-03-26T02:00:40.798 INFO:teuthology.orchestra.run.smithi003.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-26T02:00:40.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:00:40.801 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi003:/dev/vg_nvme/lv_3... 2024-03-26T02:00:40.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo nvme disconnect -n lv_3 2024-03-26T02:00:40.998 INFO:teuthology.orchestra.run.smithi003.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-26T02:00:40.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:00:41.000 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi003:/dev/vg_nvme/lv_4... 2024-03-26T02:00:41.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo nvme disconnect -n lv_4 2024-03-26T02:00:41.173 INFO:teuthology.orchestra.run.smithi003.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-26T02:00:41.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:00:41.174 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-26T02:00:41.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/scratch_devs 2024-03-26T02:00:41.201 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi055:/dev/vg_nvme/lv_1... 2024-03-26T02:00:41.201 DEBUG:teuthology.orchestra.run.smithi055:> sudo nvme disconnect -n lv_1 2024-03-26T02:00:41.351 INFO:teuthology.orchestra.run.smithi055.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-26T02:00:41.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:00:41.353 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi055:/dev/vg_nvme/lv_2... 2024-03-26T02:00:41.353 DEBUG:teuthology.orchestra.run.smithi055:> sudo nvme disconnect -n lv_2 2024-03-26T02:00:41.497 INFO:teuthology.orchestra.run.smithi055.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-26T02:00:41.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:00:41.499 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi055:/dev/vg_nvme/lv_3... 2024-03-26T02:00:41.499 DEBUG:teuthology.orchestra.run.smithi055:> sudo nvme disconnect -n lv_3 2024-03-26T02:00:41.650 INFO:teuthology.orchestra.run.smithi055.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-26T02:00:41.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:00:41.652 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi055:/dev/vg_nvme/lv_4... 2024-03-26T02:00:41.652 DEBUG:teuthology.orchestra.run.smithi055:> sudo nvme disconnect -n lv_4 2024-03-26T02:00:41.812 INFO:teuthology.orchestra.run.smithi055.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-26T02:00:41.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:00:41.814 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T02:00:41.814 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/scratch_devs 2024-03-26T02:00:41.845 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi066:/dev/vg_nvme/lv_1... 2024-03-26T02:00:41.846 DEBUG:teuthology.orchestra.run.smithi066:> sudo nvme disconnect -n lv_1 2024-03-26T02:00:41.999 INFO:teuthology.orchestra.run.smithi066.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-26T02:00:42.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:00:42.001 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi066:/dev/vg_nvme/lv_2... 2024-03-26T02:00:42.001 DEBUG:teuthology.orchestra.run.smithi066:> sudo nvme disconnect -n lv_2 2024-03-26T02:00:42.144 INFO:teuthology.orchestra.run.smithi066.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-26T02:00:42.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:00:42.145 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi066:/dev/vg_nvme/lv_3... 2024-03-26T02:00:42.145 DEBUG:teuthology.orchestra.run.smithi066:> sudo nvme disconnect -n lv_3 2024-03-26T02:00:42.296 INFO:teuthology.orchestra.run.smithi066.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-26T02:00:42.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:00:42.298 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi066:/dev/vg_nvme/lv_4... 2024-03-26T02:00:42.298 DEBUG:teuthology.orchestra.run.smithi066:> sudo nvme disconnect -n lv_4 2024-03-26T02:00:42.446 INFO:teuthology.orchestra.run.smithi066.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-26T02:00:42.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:00:42.448 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T02:00:42.448 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/scratch_devs 2024-03-26T02:00:42.481 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-26T02:00:42.493 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-26T02:00:42.494 DEBUG:teuthology.orchestra.run.smithi003:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T02:00:42.496 DEBUG:teuthology.orchestra.run.smithi055:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T02:00:42.499 DEBUG:teuthology.orchestra.run.smithi066:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T02:00:42.508 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2024-03-26T02:00:42.513 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T02:00:42.513 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2024-03-26T02:00:42.513 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 13 -24us[ -24us] +/- 94ms 2024-03-26T02:00:42.513 INFO:teuthology.orchestra.run.smithi003.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 -464us[ -465us] +/- 78ms 2024-03-26T02:00:42.513 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 13 +423us[ +423us] +/- 66ms 2024-03-26T02:00:42.513 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:00:42.513 INFO:teuthology.orchestra.run.smithi055.stderr:bash: line 1: ntpq: command not found 2024-03-26T02:00:42.518 INFO:teuthology.orchestra.run.smithi055.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T02:00:42.518 INFO:teuthology.orchestra.run.smithi055.stdout:=============================================================================== 2024-03-26T02:00:42.518 INFO:teuthology.orchestra.run.smithi055.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 13 -19us[-8680ns] +/- 94ms 2024-03-26T02:00:42.518 INFO:teuthology.orchestra.run.smithi055.stdout:^* hv02.front.sepia.ceph.com 3 6 377 11 -454us[ -443us] +/- 78ms 2024-03-26T02:00:42.518 INFO:teuthology.orchestra.run.smithi055.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 13 +427us[ +437us] +/- 66ms 2024-03-26T02:00:42.518 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:00:42.536 INFO:teuthology.orchestra.run.smithi066.stderr:bash: line 1: ntpq: command not found 2024-03-26T02:00:42.540 INFO:teuthology.orchestra.run.smithi066.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T02:00:42.540 INFO:teuthology.orchestra.run.smithi066.stdout:=============================================================================== 2024-03-26T02:00:42.540 INFO:teuthology.orchestra.run.smithi066.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 13 -19us[-7733ns] +/- 94ms 2024-03-26T02:00:42.540 INFO:teuthology.orchestra.run.smithi066.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 -464us[ -453us] +/- 78ms 2024-03-26T02:00:42.541 INFO:teuthology.orchestra.run.smithi066.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 13 +421us[ +432us] +/- 66ms 2024-03-26T02:00:42.541 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:00:42.542 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-26T02:00:42.551 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-26T02:00:42.553 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-26T02:00:42.562 DEBUG:teuthology.orchestra.run.smithi003:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T02:00:42.637 DEBUG:teuthology.orchestra.run.smithi055:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T02:00:42.710 DEBUG:teuthology.orchestra.run.smithi066:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T02:00:42.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T02:00:42.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:00:42.820 DEBUG:teuthology.orchestra.run.smithi055:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T02:00:42.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:00:42.852 DEBUG:teuthology.orchestra.run.smithi066:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T02:00:42.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:00:42.887 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-26T02:00:42.896 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-26T02:00:42.906 INFO:teuthology.task.internal:Duration was 681.969842 seconds 2024-03-26T02:00:42.907 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-26T02:00:42.916 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-26T02:00:42.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T02:00:42.919 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T02:00:42.922 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T02:00:42.960 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T02:00:42.964 INFO:teuthology.orchestra.run.smithi055.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T02:00:42.978 INFO:teuthology.orchestra.run.smithi066.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T02:00:43.430 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-26T02:00:43.430 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi003.front.sepia.ceph.com 2024-03-26T02:00:43.430 DEBUG:teuthology.orchestra.run.smithi003:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T02:00:43.452 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi055.front.sepia.ceph.com 2024-03-26T02:00:43.452 DEBUG:teuthology.orchestra.run.smithi055:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T02:00:43.474 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi066.front.sepia.ceph.com 2024-03-26T02:00:43.474 DEBUG:teuthology.orchestra.run.smithi066:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T02:00:43.497 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-26T02:00:43.498 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T02:00:43.500 DEBUG:teuthology.orchestra.run.smithi055:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T02:00:43.516 DEBUG:teuthology.orchestra.run.smithi066:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T02:00:43.570 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-26T02:00:43.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T02:00:43.573 DEBUG:teuthology.orchestra.run.smithi055:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T02:00:43.586 DEBUG:teuthology.orchestra.run.smithi066:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T02:00:43.757 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-26T02:00:43.767 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-26T02:00:43.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T02:00:43.794 DEBUG:teuthology.orchestra.run.smithi055:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T02:00:43.821 DEBUG:teuthology.orchestra.run.smithi066:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T02:00:43.852 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-26T02:00:43.862 DEBUG:teuthology.orchestra.run.smithi003:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T02:00:43.865 DEBUG:teuthology.orchestra.run.smithi055:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T02:00:43.867 DEBUG:teuthology.orchestra.run.smithi066:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T02:00:43.887 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = core 2024-03-26T02:00:43.890 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = core 2024-03-26T02:00:43.921 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern = core 2024-03-26T02:00:43.952 DEBUG:teuthology.orchestra.run.smithi003:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T02:00:43.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:00:43.983 DEBUG:teuthology.orchestra.run.smithi055:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T02:00:43.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:00:43.997 DEBUG:teuthology.orchestra.run.smithi066:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T02:00:44.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:00:44.012 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-26T02:00:44.022 INFO:teuthology.task.internal:Transferring archived files... 2024-03-26T02:00:44.023 DEBUG:teuthology.misc:Transferring archived files from smithi003:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621927/remote/smithi003 2024-03-26T02:00:44.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T02:00:44.061 DEBUG:teuthology.misc:Transferring archived files from smithi055:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621927/remote/smithi055 2024-03-26T02:00:44.062 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T02:00:44.100 DEBUG:teuthology.misc:Transferring archived files from smithi066:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621927/remote/smithi066 2024-03-26T02:00:44.100 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T02:00:44.141 INFO:teuthology.task.internal:Removing archive directory... 2024-03-26T02:00:44.141 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T02:00:44.143 DEBUG:teuthology.orchestra.run.smithi055:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T02:00:44.145 DEBUG:teuthology.orchestra.run.smithi066:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T02:00:44.191 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-26T02:00:44.203 INFO:teuthology.task.internal:Not uploading archives. 2024-03-26T02:00:44.204 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-26T02:00:44.286 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-26T02:00:44.286 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T02:00:44.289 DEBUG:teuthology.orchestra.run.smithi055:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T02:00:44.291 DEBUG:teuthology.orchestra.run.smithi066:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T02:00:44.300 INFO:teuthology.orchestra.run.smithi003.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 02:00 /home/ubuntu/cephtest 2024-03-26T02:00:44.303 INFO:teuthology.orchestra.run.smithi055.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 02:00 /home/ubuntu/cephtest 2024-03-26T02:00:44.306 INFO:teuthology.orchestra.run.smithi066.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 02:00 /home/ubuntu/cephtest 2024-03-26T02:00:44.307 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-26T02:00:44.318 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-26T02:00:44.338 INFO:teuthology.nuke:Checking targets against current locks 2024-03-26T02:00:44.362 DEBUG:teuthology.nuke:shortname: smithi003 2024-03-26T02:00:44.362 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T02:00:44.384 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621927', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 01:37:45.880175', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHu32xGMBqAWvAJXCYWtSJyY3WoRcqOMPWX/KkOfrndVMku6Z42iwzgidh532QIh8HHdSHr1op51fO/tF68lt3M='} 2024-03-26T02:00:44.389 DEBUG:teuthology.nuke:shortname: smithi055 2024-03-26T02:00:44.389 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T02:00:44.411 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621927', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 01:37:45.881190', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPxwSXOgZgN7upLALulxHsKZY0dvBNg4rRvp3c2TPaCOIzHvdCHGkYTSgMLXT3dxjX94G/D8TiwKLT0eVjD7XLE='} 2024-03-26T02:00:44.416 DEBUG:teuthology.nuke:shortname: smithi066 2024-03-26T02:00:44.417 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T02:00:44.443 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi066.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621927', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 01:37:45.878916', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBADvA3mJIbyPUTGnJwZlNnBhoxmejaDAjEo1Kw5H5arYF5+cVTZlYPnORJi2jTlzc0yAVSkY3iiJPrDvuKTg26M='} 2024-03-26T02:00:44.450 INFO:teuthology.orchestra.console.smithi003:Power off 2024-03-26T02:00:44.450 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T02:00:44.474 DEBUG:teuthology.orchestra.console.smithi003:power off output: Chassis Power Control: Down/Off 2024-03-26T02:00:44.475 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:00:44.494 INFO:teuthology.orchestra.console.smithi055:Power off 2024-03-26T02:00:44.494 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T02:00:44.506 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2024-03-26T02:00:44.518 DEBUG:teuthology.orchestra.console.smithi055:power off output: Chassis Power Control: Down/Off 2024-03-26T02:00:44.518 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:00:44.534 INFO:teuthology.orchestra.console.smithi066:Power off 2024-03-26T02:00:44.534 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T02:00:44.543 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-03-26T02:00:44.557 DEBUG:teuthology.orchestra.console.smithi066:power off output: Chassis Power Control: Down/Off 2024-03-26T02:00:44.557 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:00:44.580 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is on 2024-03-26T02:00:48.508 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:00:48.544 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:00:48.581 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:00:48.619 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2024-03-26T02:00:48.654 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-03-26T02:00:48.691 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is on 2024-03-26T02:00:52.624 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:00:52.654 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:00:52.692 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:00:52.734 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is off 2024-03-26T02:00:52.764 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-26T02:00:52.802 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is off 2024-03-26T02:00:52.835 INFO:teuthology.orchestra.console.smithi003:Power off completed 2024-03-26T02:00:52.865 INFO:teuthology.orchestra.console.smithi055:Power off completed 2024-03-26T02:00:52.902 INFO:teuthology.orchestra.console.smithi066:Power off completed 2024-03-26T02:00:53.007 INFO:teuthology.run:Summary data: description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} duration: 681.9698424339294 failure_reason: '"2024-03-26T01:56:48.619706+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-26T02:00:53.007 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T02:00:53.083 INFO:teuthology.run:FAIL